On colour – a visual list

On colourSome referencesA visual list

Books

On the web

Papers

(e.g., spatial frequency, orientation, motion, depth)within a local cortical region. With respect to colorvision per se, the primary processing involves separ-ating color and luminance information, and furtherseparating changes due to the illuminant from thosedue to visual objects, by lateral interactions over largeregions.

To separate luminance and color information, theoutputs of P

ccells are combined in two different ways.

When their outputs are summed in one way, theluminance components to their responses sum and thecolor components cancel. Summed in a differentcombination, the color components sum and theluminance components cancel. Consider a striatecortex cell that combines inputs from one or more­L

oand ­M

ocells in a region. The cortical cell would

respond to luminance variations but not to colorvariations, since the neurons providing its inputs bothfire to luminance increments in the RF center and todecrements in the surround, but the color organi-zations of its inputs are opposite to each other (onebeing L-M and the other M-L). Combined with inputfrom a ­S

ocell, this would produce a V1 cell that fires

to white (light increments) and inhibits to black (lightdecrements) but does not respond to pure colorvariations. This is represented in the top row of Fig.1C. However, a V1 cell receiving inputs from both­L

oand ®M

ocells, or from both ­M

oand ®L

ocells

(columns in Fig. 1C), would respond to color changesbut not to luminance variations since their colorresponses would add, but their luminance RFs, whichare opposite to each other, would cancel. This organi-zation by itself would produce L-M color cells thatwould fire to so-called warm colors (red and yellow)and inhibit to cool colors (blue and green). M-L cellswould fire to cool colors and inhibit to warm colors.As shown in Fig. 1C, the further addition of ­S

oor

®So

cells can split these classes into separate redand yellow, and separate blue and green systems,respectively.

All of the primary visual information is passedthrough V1, but subsequent visual areas are partiallyspecialized for the further analysis of various differentfunctional aspects of vision. One later visual area (V4)is crucially involved with color perception. Individualswith localized V4 lesions can still discriminate objectson the basis of their color variations, but they reportthat the objects now appear to have no hue, as ifviewed on a black-white television screen. There is alsoa report of one case with the reverse loss: a patient whocould see colored but not black-white objects.

11. Color Appearance

The appearance of a color can be specified by valuesalong just three perceptual dimensions known as hue,saturation and brightness. Hue refers to the character-istic described by such color names as red, yellow,green, and blue. Saturation refers to the extent to

which the stimulus differs perceptually from a purelyachromatic (i.e., white, gray, black) axis. The thirddimension is brightness or lightness. That our per-ceptual space is three-dimensional reflects the basictrichromacy of vision.

A normal observer can describe the hue of any light(disregarding surface characteristics) by using one ormore of only four color names (red, yellow, green, andblue). These so-called unique hues form two opponentpairs, red–green and blue–yellow. Red and greennormally cannot be seen in the same place at the sametime; if unique red and unique green lights are added inappropriate proportions, the colors cancel and onesees a neutral gray. Orange can be seen as a mixture ofred and yellow, and purple as a mixture of red andblue, but there is no color seen as a red–green mixture(or as a blue–yellow mixture). This perceptual op-ponency is also reflected in color contrast. Red caninduce the appearance of green into neighboringregions, and after staring at a red surface one sees agreen after-image. The yellow–blue opponent pairproduces similar effects. It was these perceptualcharacteristics of color that led Ewald Hering in thenineteenth century to propose that the various colorsystems were not independent but rather that colorwas processed in a spectrally opponent organization,an idea which has since been amply verified in thepresence, discussed above, of spectrally-opponent cellsin the path from receptors to the cortex.

See also: Color Vision Theory; Vision, Low-levelTheory of; Vision, Psychology of; Visual Perception,Neural Basis of; Visual System in the Brain

Bibliography

De Valois R L, De Valois R L 1988 Spatial Vision. OxfordUniversity Press, New York

Hurvich L M 1981 Color Vision. Sinauer Press, Sunderland, MAKaiser P K, Boynton R M 1996 Human Color Vision. Optical

Society of America, Washington, DCNeitz J, Neitz M 1998 Molecular genetics and the biological

basis of color vision. In: Backhaus W G S, Kliegl R, WernerJ S (eds.) Color Vision. Walter de Gruyter, Berlin, pp. 101–19

Spillmann L, Werner J S 1990 Visual Perception: The Neuro-physiological Foundations. Academic Press, New York

K. K. De Valois and R. L. De Valois

Color Vision Theory

Color vision is the ability to distinguish and identifylights and objects on the basis of their spectralproperties. This entry presents several key topics thatunderlie current theories of human color vision. Theseare trichromacy, color opponency, adaptation, andcolor constancy.

2256

Color Vision

1. Introduction

Information about color is transformed as it flowsfrom the stimulus through the initial stages of thehuman visual system. At each image location, thecolor stimulus is specified by the amount of power itcontains at each wavelength. The classic color match-ing experiment shows that the normal human visualsystem is trichromatic: only three dimensions ofspectral variation are coded by the visual system. Thebiological basis of normal trichromacy is that theretina contains three classes of cone photopigment.After the initial encoding of light by the cones, furtherprocessing occurs. Two aspects of this processing areparticularly important. First, signals from three classesof cones are recombined to form a luminance and twocolor opponent channels. Second, there is adaptivesignal regulation that keeps neural signals within theiroperating range and stabilizes the appearance ofobjects across changes of illumination.

2. Trichromacy

2.1 Color Matching

The physical property of light relevant for color visionis the spectral power distribution. A light’s spectralpower distribution specifies the amount of power itcontains at each wavelength in the visible spectrum,often taken to lie roughly between 400 and 700 nm. Inpractice, spectral power distributions are measured atdiscrete sample wavelengths. Let the measured powervalues be denoted by b

",…, b

Nλwhere Nλ denotes the

number of sample wavelengths. Then the vector

b¯

A

B

b"

]

bNλ

C

D

(1)

provides a compact representation of the spectralpower distribution. Use of a vector representation forspectral quantities facilitates a variety of colorimetriccomputations (e.g., Brainard 1995). Wavelengthsample spacings between 1 and 10nm are typical.

Trichromacy is demonstrated by the basic colormatching experiment (Wandell 1995, Brainard 1995).In this experiment, an observer views a bipartite field.One side of the field contains a test light. This light isexperimentally controlled and can have an arbitraryspectral power distribution. On the other side of thefield is the matching light. This consists of the weightedmixture of three primary lights. Each primary has afixed relative spectral power distribution, but itsoverall intensity in the mixture can be controlled bythe observer. The observer’s task is to adjust the

primary intensities until the mixture has the samecolor appearance as the test light. The primaries usedin the experiment are chosen to be independent, sothat no weighted mixture of any two produces a matchto the third.

Because the matching light is constrained to be aweighted mixture of three primaries, it will notgenerally be possible for the observer to make the testand matching lights physically identical. For manytest lights, however, the observer can adjust thematching light so that it appears identical to the testlight even though the two differ physically. For sometest lights, no choice of primary intensities will afforda match. In these cases one or more of the primariescan be mixed with the test light and primary intensitiesfound so that the primary}test mixture matches themixture of the remaining primaries. A useful descrip-tive convention for the color matching experiment isto assign a negative intensity to any primary that mustbe mixed with the test to make a match. Given thisconvention, any test light can be matched by a mixtureof three independent primaries.

The color matching experiment is an empiricalsystem. Given a test light described by a vector b, theexperiment returns a vector

t¯

A

B

t"

t#

t$

C

D

(2)

whose entries are the individual primary intensities.When the primaries are scaled by these intensities andmixed, a match to the test light is created. The vectort specifies what are called the tristimulus coordinatesof the light b. A theory of color matching should let uspredict t for any test light b, given the spectral powerdistributions of the primary lights.

As an empirical generalization, the color matchingsystem is a linear system (e.g., Wyszecki and Stiles1982, Brainard 1995, Wandell 1995). That is, if wehave two test lights b

"and b

#with tristimulus

coordinates t"

and t#, then any weighted mixture

(a"b"­a

#b#) of the two test lights has tristimulus

coordinates given by the corresponding mixture(a

"t"­a

#t#). In these vector expressions, multiplication

of a vector (e.g., b") by a scalar (e.g., a

") consists of

multiplying each entry of the vector by the scalar,while addition of two vectors (e.g., a

"b"

and a#b#)

consists of adding the corresponding entries of the twovectors.

The linearity of color matching makes it possible topredict the match that will be made to any test light onthe basis of a relatively small number ofmeasurements.Consider the set of monochromatic lights with unitpower. If Nλ wavelength samples are used in theunderlying representation, then there are Nλ suchlights and we can denote their spectral representationsby c

", c

#, …, c

Nλ. Each of the ci has a 1 as its ith entry

2257

Color Vision Theory

and zeros elsewhere. Note that any light b may bethought of as a weighted mixture of monochromaticlights, so that b¯3

ibici where b

iis the ith entry of b.

Let the vectors tispecify the tristimulus coordinates

of the monochromatic lights ci. The linearity of color

matching then tells us that the tristimulus coordinatesof any light b are given by t¯3

ibiti.

A set of tristimulus values timeasured for mono-

chromatic lights ci

is referred to as a set of colormatching functions. Although these are often plottedas a function of wavelength, they do not represent thespectral power distributions of lights. The colormatching functions may be specified by a single matrix

T¯A

Bt"t#t$It

Nλ

C

D(3)

whose Nλ columns consist of the individual tristimuluscoordinate vectors t

i. This specification allows com-

putation of tristimulus coordinates from spectralpower distributions through simple matrix multipli-cation:

t¯Tb. (4)

Both tristimulus values and color matching functionsare defined with respect to the primaries chosen for theunderlying color matching experiment. The Com-mission Internationale de l’Eclairage (CIE) has stan-dardized a system for color representation based onthe ideas outlined above. The CIE system is widelyused to specify color stimuli andmany sources describeit in detail (e.g., Wyszecki and Stiles 1982, Brainard1995, Kaiser and Boynton 1996).

The advantage of using tristimulus coordinates todescribe color stimuli is that they provide a morecompact and tractable description than a descriptionin terms of wavelength. Tristimulus coordinates arecompact precisely because they do not preserve physi-cal differences that are invisible to the human visualsystem. The representational simplification affordedby tristimulus coordinates is extremely valuable forstudying processing that occurs after the initial encod-ing of light. On the other hand, it is important toremember that the standard tristimulus represen-tations (e.g., the CIE system) are based on matchesmade by a typical observer looking directly at a smallstimulus at moderate to high light levels. Theserepresentations are not necessarily appropriate forapplications involving some individual observers, non-human color vision, or color cameras (e.g., Wyszeckiand Stiles 1982, Brainard 1995).

2.2 Biological Basis of Color Matching

The color matching experiment is agnostic about thebiological mechanisms that underlie trichromacy. It isgenerally accepted, however, that trichromacy typi-cally arises because color vision is mediated by threetypes of cone photoreceptor. Direct physiologicalmeasurements of individual primate cones support

this hypothesis (see Wandell 1995, Rodieck 1998).First, the responses of individual cones depend onlyon the rate at which photopigment molecules areisomerized by the absorption of light quanta; once theintensity of two lights has been adjusted so that theyproduce the same isomerization rates, the cone re-sponse does not distinguish the two lights. This idea isreferred to as the principle of univariance. Second,individual cones may be classified into one of threedistinct types, each with a characteristic spectralsensitivity. The spectral sensitivity is proportional tothe probability that light quanta of different wave-lengths will isomerize a molecule of the cone’s photo-pigment. The three types of cones are often referred toas the long- (L), middle- (M), and short- (S) wave-length-sensitive cones. If an observer has only threetypes of cones, each of which obeys the principle ofunivariance, two physically distinct lights that producethe same isomerization rates for all three classes ofcones will be indistinguishable to the visual system.Quantitative comparison confirms that color matchesset by a standard observer (defined as the average ofmatches set by many individual observers) are wellpredicted by the equations of isomerization rates inthe L-, M-, and S-cones.

As described above, trichromacy occurs for mostobservers because their retinas contain cones withthree classes of photopigments. Genetic consider-ations, however, indicate that some individuals haveretinas containing four classes of cone photopigments(Sharpe et al. 1999). Either these individuals aretetrachromatic (mixture of four primaries required tomatch any light) or else their trichromacy is mediatedby information lost after quantal absorption. Inaddition, some human observers are dichromatic (onlytwo primaries must be mixed to make a match to anylight.) Most cases of dichromacy occur because onephotopigment is missing (Sharpe et al. 1999, Neitz andNeitz 2000).

An alternative to using tristimulus coordinates torepresent the spectral properties of lights is to use conecoordinates. These are proportional to the isomeriz-ation rates of the three classes of cone photopigments.The three dimensional vector

q¯

A

B

qL

qM

qS

C

D

(5)

specifies cone coordinates where qL, q

M, and q

Sdenote

the isomerization rates of the L-, M-, and S-conephotopigments respectively. It can be shown (e.g.,Brainard 1995) that cone coordinates and tristimuluscoordinates are related by a linear transformation, sothat

q¯Mtqt (6)

where Mtq

is an appropriately chosen 3 by 3 matrix.

2258

Color Vision Theory

Computation of cone coordinates from light spectrarequires estimates of the cone spectral sensitivities.For each cone class, these specify the isomerizationrates produced by monochromatic lights of unitpower. The sensitivities may be specified in matrixform as

S¯

A

B

sL

sM

sS

C

D

(7)

where each row of the matrix is a vector whose entriesare the spectral sensitivities for one cone class at thesample wavelengths. Given S, cone coordinates arecomputed from the spectral power distribution of alight as

q¯Sb (8)

Because Eqns. (4), (6), and (8) hold for any lightspectrum b, it follows that

S¯MtqT (9)

Current estimates of human cone spectral sensitivitiesare obtained from color matching data using Eqn. (9)together with a variety of considerations that putconstraints on the matrix M

tq(Stockman and Sharpe

1999).

3. Postabsorption Processing

Color vision does not end with the absorption of lightby cone photopigments. Rather, the signals thatoriginate with the absorption of light are transformedas they propagate through neurons in the retina andcortex. Two ideas dominate models of this post-absorption processing. The first is color opponency:signals from different cone types are combined in anantagonistic fashion to produce the visual represen-tation at a more central site. The second idea isadaptation: the relation between the cone coordinatesof a light and its central visual representation is notfixed but depends instead on the context in which thelight is viewed. Section 3.1 treats opponency, whileSect. 3.2 treats adaptation.

3.1 Opponency

Direct physiological measurements of the responses ofneurons in the primate retina support the general ideaof opponency (e.g., Dacey 2000). These measurementsreveal, for example, that some retinal ganglion cellsare excited by signals from L-cones and inhibited bysignals from M-cones. One suggestion about why thisoccurs is that it is an effective way to code the conesignals for transmission down the optic nerve (seeWandell 1995).

A possible approach to understanding post-absorp-tion processing is to keep the modeling close to theunderlying anatomy and physiology and to character-ize what happens to signals at each synapse in theneural chain between photoreceptors and some site invisual cortex. The difficulty is that it is not presentlypossible to cope with the complexity of actual neuralprocessing. Thus many color theorists have attemptedto step back from the details and develop moreabstract descriptions of the effect of neural processing.Models of this sort are often called mechanisticmodels. These models generally specify a transform-ation between the quantal absorption rates q elicitedby a stimulus and a corresponding visual represen-tation u postulated to exist at some central site. Theidea is to choose a transformation so that (a) the colorappearance perceived at a location may be obtaineddirectly from the central representation correspondingto that location and (b) the discriminability of twostimuli is predictable from the difference in theircentral representations.

Most mechanistic models assume that signals fromthe cones are combined additively to produce signalsat three postreceptoral sites. Two of these sites carryopponent signals. These are often referred to as thered-green (RG) and blue-yellow (BY) signals. A thirdsite carries a luminance (LUM) signal, which is notthought to be opponent. If we take

u¯

A

B

uLUM

uRG

uBY

C

D

(10)

to be a three-dimensional vector with entries given bythe LUM, RG, and BY signals, then the additiverelation between cone coordinates q and the visualrepresentation u may be expressed in matrix form:

u¯Moq (11)

Many (but not all) detailed models take LUM to be aweighted sum of L- and M-cone signals, RG to bea weighted difference between the L- and M-conesignals, and BY to be a weighted difference betweenthe S-cone signal and a weighted sum of the L- and M-cone signals. In these models M

owould have the form

Mo¯

A

B

m""

m"#

0

m#"

®m##

0

®m$"

®m$#

m$$

C

D

(12)

where all of the mij

are positive scalars representinghow strongly one cone class contributes to the signal atone post-receptoral site.

Considerable effort has been devoted to establishingwhether the linear form for the mapping between qand u is appropriate, and if so, what values should be

2259

Color Vision Theory

used for the mij. Several types of experimental evidence

have been brought to bear on the question.As an example, one line of research argues that four

color perceptions, those of redness, greenness, blue-ness, and yellowness, have a special psychologicalstatus, in that any color experience may be intuitivelydescribed in terms of these four basic perceptions.Thus orange may be naturally described as reddish-yellow and aqua as greenish-blue. In addition, bothintrospection and color scaling experiments suggestthat the percepts of redness and greenness are mutuallyexclusive so that both are not experienced simul-taneously in response to the same stimulus, andsimilarly for blueness and yellowness (e.g., Hurvichand Jameson 1957, Abramov and Gordon 1994).Given these observations, it is natural to associate theRG signal with the amount of redness or greennessperceived in a light (redness if the signal is positive,greenness if it is negative, and neither red nor green ifit is zero) and the BY signal with the amount ofblueness or yellowness. Judgments of the four fun-damental color perceptions, obtained either throughdirect scaling (e.g., Abramov and Gordon 1994) orthrough a hue cancellation procedure (e.g., Hurvichand Jameson 1957), are then used to deduce theappropriate values of the m

ijin the second and third

rows of Mo. When this framework is used, the entries

for the first row of Mo, corresponding to the LUM

signal, are typically established through other meanssuch as flicker photometry (e.g., Kaiser and Boynton1996).

Other approaches to studying the opponent trans-formation include analyzing measurements of thedetection and discrimination of stimuli (e.g., Wyszeckiand Stiles 1982, Kaiser and Boynton 1996, Eskew,et al. 1999, Wandell 1999), and measurements of howthe color appearance of lights is affected by the contextin which they are viewed (e.g., Webster 1996). In partbecause of a lack of quantitative agreement in theconclusions drawn from different paradigms, there iscurrently not much consensus about the details of thetransformation between q and u. One of the majoropen issues in color theory remains how to extend thesimple linear model described above so that it accountsfor a wider range of results.

3.2 Adaptation

Figure 1 illustrates a case where the same light has avery different color appearance when seen in twodifferent contexts. The figure shows two disk-annulusstimulus configurations. The central disk is the same ineach configuration, but the appearance of the twodisks is quite different. To explain this and othercontext effects, mechanistic models assume that at anygiven time and image location, the relation betweenthe quantal absorption rates q and the visual rep-resentation u depends on the quantal absorption rates

Figure 1A color context effect. The figure illustrates the colorcontext effect known as simultaneous contrast. The twocentral disks are physically the same but appeardifferent. The difference in appearance is caused by thefact that each disk is seen in the context of a differentannular surround. This figure is best viewed in color.A color version is available in the on-line version of theEncyclopedia

at other locations and at preceding times. To help fixideas, it is useful to restrict attention to the disk-annulus configuration. For this configuration, thevisual representation of the disk may be written as

ud¯ f (q

d; q

a,}) (13)

where udis the visual response to the disk, q

dand q

aare

the cone coordinates of the disk and annulus re-spectively, and } represents other contextual variablessuch as the size of the disk and annulus and anytemporal variation in the stimulus. Clearly, f( ) mustincorporate the sort of transformation described bythe matrix M

oin Sect. 3.1 above.

As was the case with the discussion of opponencyabove, there is not wide agreement about how best tomodel adaptation. A reasonable point of departure isa cone-specific affine model. In this model, the visualrepresentation u of a light is related to its conecoordinates q through an equation of the form

u¯Mo(D

"q®q

") (14)

where Mois as in Eqn. (12) and

D"¯

A

B

gL"

0 0

0 gM"

0

0 0 gS"

C

D

, q"¯

A

B

qL"

qM"

qS"

C

D

(15)

In this formulation, the g’s on the diagonals of D"

characterize multiplicative adaptation that occurs at acone-specific site in visual processing, before signalsfrom separate cone classes are combined. The entriesof the vector q

"characterize subtractive adaptation.

Equation (14) is written in a form that implies that thesubtractive adaptation also occurs at a cone-specific

2260

Color Vision Theory

site. The entries of D"

and q"

depend on the conecoordinates q

aof the annulus as well as on spatial and

temporal variables characterized by }. Note that thecone-specific affine model is a generalization of theidea that the visual representation consists of acontrast code.

Asymmetric matching may be used to test theadaptation model of Eqn. (14). In an asymmetricmatching experiment, an observer adjusts a matchstimulus seen in one context so that it appears to havethe same color as a test stimulus seen in anothercontext. More concretely, consider Fig. 1. In thecontext of this figure, an asymmetric matching ex-periment could be conducted where the observer wasasked to adjust the central disk on the right so that itmatched the appearance of the central test disk on theleft. Suppose such data are collected for a series ofN test disks with cone coordinates q

ti. Denote the

cone coordinates of the matches by qmi

. Within themechanistic framework, the corresponding visual rep-resentations u

tiand u

mishould be equal. If Eqn. (14)

provides a good description of performance then

Mo(D

m"qmi

®qm"

)¯Mo(D

t"qti®q

t")

5 qmi

¯D−"m"

(Dt"qti®q

t"­q

m")

5 qmi

¯Dtm

qti®q

tm

(16)

where Dtm

¯D−"m"

Dt"

and qtm

¯D−"m"

(qt"®q

m"). This

prediction may be checked by finding the diagonalmatrix D

tmand vector q

tmthat provide the best fit to

the data and evaluating the quality of the fit. Tests ofthis sort indicate that the cone specific affine modelaccounts for much of the variance in asymmetricmatching data, both for the disk annulus configuration(Wandell 1995, 1999) and for more complex stimuli(Brainard and Wandell 1992). Nonetheless, there areclear instances for which Eqn. (16) does not givea complete account of asymmetric matching (e.g.,Delahunt and Brainard 2000) and other color ap-pearance data (e.g., Webster 1996, Mausfeld 1998,D’Zmura and Singer 1999).

The cone-specific affine model may also be testedagainst psychophysical data on the detection anddiscrimination of colored lights. Here again the modelprovides a reasonable point of departure but fails indetail (e.g., Eskew et al. 1999).

To extend the cone specific affine model, varioustheorists have suggested the need for adaptation at asecond site (after signals from separate cone classeshave been combined) and for the inclusion of non-linearities in the relation between q and u (seereferences cited in the previous two paragraphs). Anadditional open question concerns how the entries ofD

"and q

"are determined by the viewing context

(e.g., Brainard and Wandell 1992, Delahunt andBrainard 2000).

4. Color Constancy

The discussion so far has focussed on how the visualsystem represents and processes the spectrum of lightthat enters the eye. This is natural, since light is theproximal stimulus that initiates color vision. On theother hand, we use color primarily to name objects.The spectrum of the light reflected to the eye from anobject depends both on an intrinsic property of theobject, its surface reflectance function, and on extrinsicfactors, including the spectral power distribution ofthe illuminant and how the object is oriented relativeto the observer.

Given that the light reflected to the eye varies withthe illuminant and viewing geometry, how is it thatcolor is a useful psychological property of objects? Theanswer is that the visual system processes the retinalimage to stabilize the color appearance of objectsacross changes extrinsic to the object (e.g., changes inthe spectrum of the illuminant). This stabilizationprocess is called color constancy.

Color constancy is closely linked to the phenom-enon of adaptation described above (Maloney 1999).Indeed, quantitative models of color constancy gen-erally incorporate the same idea that underlies mech-anistic models of visual processing: at some central sitethere is a visual representation u that correlates withcolor appearance. To stabilize this representationagainst changes in illumination, it is supposed that therelation between the quantal absorption rates q elicitedby the light reflected from an object and the visualrepresentation u depends on the scene in which theobject is viewed. In the case of color constancy, theemphasis has been on how the visual system processesthe retinal image so that the transformation between qand u has the effect of compensating for the variationin the light reflected to the eye caused by changes ofillumination and viewing geometry. Psychophysicaldata on the color appearance of objects viewed underdifferent illuminants are often well-modeled by trans-formations consistent with Eqn. (14) (e.g., Brainardand Wandell 1992).

The central theoretical question of color constancyis how the visual system can start with image data andfactor it into an accurate representation of the surfacesand illuminants in the scene. This question has receivedextensive treatment, at least for simple scenes. A briefintroduction to this literature on computational colorconstancy follows.

4.1 Computational Color Constancy

Consider a scene consisting of diffusely illuminatedflat, matte surfaces. For such scenes, the spectrum breflected to the eye from each surface is given by thewavelength-by-wavelength product of the spectralpower distribution of the illuminant e and the surfacereflectance function s. The surface reflectance functionspecifies, at each sample wavelength, the fraction of

2261

Color Vision Theory

incident light reflected to the eye. The informationabout b coded by the visual system is its conecoordinates, which may be computed as

q¯Sb¯S diag(e)s (17)

where the function diag( ) returns a square diagonalmatrix with its argument placed along the diagonal.Clearly e and s are not uniquely determined fromknowledge of q: without additional constraints thecolor constancy problem is underdetermined. For-tunately the spectra of naturally occurring illuminantsand surfaces are not arbitrary. Although the physicalprocesses that constrain these spectra are not wellunderstood, analyses of measurements of both illumi-nants and surfaces shows that their spectra are welldescribed by small-dimensional linear models (seeBrainard 1995, Maloney 1999).

Consider surface reflectances. It is possible to definethree fixed basis functions so that naturally occurringsurface reflectances are reasonably well approximatedby a linear combination of these basis functions. Thusfor any surface, we have

sEws"bs"­w

s#bs#­w

s$bs$

(18)

where bs", b

s#, and b

s$are the spectra of the basis

functions and ws", w

s#, and w

s$are scalar weights

that provide the best approximation of s within thelinear model. Eqn. (18) may be rewritten as

sEBsw

s(19)

where the three columns of matrix Bscontain the basis

functions and the vector ws

contains the scalarweights.

When the surface reflectance functions lie withina three-dimensional linear model Eqn. (17) mayinverted, once an estimate e## of the illuminant hasbeen obtained (see below for discussion of illuminantestimation.) Start by rewriting Eqn. (17) as:

q¯ [S diag(e# )Bs] w

s¯Le# ws

(20)

where Le# is a three-by-three matrix that depends on theilluminant estimate. This matrix may be inverted usingstandard methods to yield an estimate of w

s:

wWs¯L−"

eW q (21)

The estimate may then be used together with Eqn. (19)to estimate the surface reflectance function.

Many computational color constancy algorithmsassume a linear model constraint for surface reflec-tance functions. This reduces the constancy problemto finding an estimate of the illuminant to plug intoEqn. (20). For illustrative purposes, an algorithmdue to Buchsbaum (1980) is described here. InBuchsbaum’s algorithm, two additional assumptionsare added. The first is that a three-dimensional

linear model also describes illuminant spectral powerdistributions, so that

eEBew

e(22)

The second is that the spatial average of the surfacereflectance functions (s- ) is the same in all scenes andknown. These additional constraints imply that

qa ¯ [S diag(sa )Be]wW

e¯L

sawW

e(23)

where q- is the spatial average of the quantal absorp-tion rates and Ls- is a known three-by-three matrix.Inverting Eqn. 23 yields an estimate for the illuminante# ¯B

ew

eW . This estimate is then used to provide the

matrix L−"eW to be used in Eqn. (21).

Buchsbaum’s algorithm shows how the addition ofappropriate assumptions allows solution of thecomputational color constancy problem.The difficultywith Buchsbaum’s algorithm is that its assumptionsare too restrictive. In particular, it seems unlikely thatthe spatial average of surface reflectances is constantacross scenes, nor do real scenes consist of diffuselyilluminated flat, matte surfaces. Subsequent work hasfocused on ways to provide reasonable estimates ofthe illuminant and surface reflectances under othersets of assumptions (e.g., Maloney 1999).

4.2 Computational Color Constancy and HumanPerformance

How does the computational work relate to humanperformance? This question has not yet been resolved,but it seems appropriate to close with a few obser-vations. First, the estimated linear model weights ofEqn. (21) may be associated with the mechanismresponses u discussed in Sect. 2. In both types oftheory, these quantities represent the visual responsecomputed from the quantal absorption rates, and bothare meant to allow direct prediction of appearance. Inthe mechanistic approach, one considers a series oftransformations whose form is derived from experi-ments with simple stimulus configurations. In thecomputational approach, the form of the transform-ation is derived from consideration of the problemcolor vision is trying to solve. In both cases, however,the emphasis is on finding the appropriate parametricform of the transformation and on understanding howthe parameters are set as a function of the image data.The connection between the two approaches is dis-cussed in more detail by Maloney (1999).

The value of the computational approach to under-standing human vision depends on how accurately thetransformations it posits may be used to predict theappearance of stimuli measured in psychophysicalexperiments. There have been only a few empiricalcomparisons of this sort to date. These comparisonsdo, however, indicate that the computational ap-proach shows promise for advancing our understand-

2262

Color Vision Theory

ing of human color vision (Bloj, Kersten, and Hurlbert1999, Brainard, Kraft, and Longre 2001).

See also: Color Vision; Psychophysical Theory andLaws, History of; Psychophysics; Vision, Low-levelTheory of; Vision, Psychology of; Visual Perception,Neural Basis of; Visual System in the Brain

Bibliography

Abramov I, Gordon J 1994 Color appearance: on seeing red—oryellow, or green, or blue. Annual Re�iew of Psychology 45:451–85

Bloj M G, Kersten D, Hurlbert A C 1999 Perception of three-dimensional shape influences colour perception throughmutual illumination. Nature 402: 877–9

Brainard D H 1995 Colorimetry. In: Bass M (ed.) Handbook ofOptics: Volume 1. Fundamentals, Techniques, and Design.McGraw-Hill, New York, pp. 26.1–26.54

Brainard D H, Brunt W A, Speigle J M 1997 Color constancy inthe nearly natural image. 1. Asymmetric matches. Journal ofthe Optical Society of America A 14: 2091–110

Brainard D H, Kraft J M, Longe' re P 2001 Color constancy:developing empirical tests of computational models. In:Mausfeld R, Heyer D (eds.) Colour Perception: From Light toObject. Oxford University Press, Oxford, UK

Brainard D H, Wandell B A 1992 Asymmetric color-matching:How color appearance depends on the illuminant. Journal ofthe Optical Society of America A 9(9): 1433–48

Buchsbaum G 1980 A spatial processor model for object colourperception. Journal of the Franklin Institute 310: 1–26

D’Zmura M, Singer B 1999 Contrast gain control. In:Gegenfurtner K, Sharpe L T (eds.) Color Vision: FromGenes to Perception. Cambridge University Press, Cambridge,UK, pp. 369–85

Dacey D M 2000 Parallel pathways for spectral coding inprimate retina. Annual Re�iew of Neuroscience 23: 743–75

Delahunt P B, Brainard D H 2000 Control of chromaticadaptation: Signals from separate cone classes interact. VisionResearch 40: 2885–903

Eskew R T, McLellan J S, Giulianini F 1999 Chromatic de-tection and discrimination. In: Gegenfurtner K, Sharpe L T(eds.) Color Vision: From Genes to Perception. CambridgeUniversity Press, Cambridge, UK, pp. 345–68

Hurvich L M, Jameson D 1957 An opponent-process theory ofcolor vision. Psychological Re�iew 64(6): 384–404

Kaiser P K, Boynton R M 1996 Human Color Vision, 2nd edn.Optical Society of America, Washington, DC

Maloney L T 1999 Physics-based approaches to modelingsurface color perception. In: Gegenfurtner K, Sharpe L T(eds.) Color Vision: From Genes to Perception. CambridgeUniversity Press, Cambridge, UK, pp. 387–416

Mausfeld R 1998 Color perception: From Grassman codes to adual code for object and illumination colors. In: BackhausW G K, Kliegl R, Werner J S (eds.) Color Vision—Pers-pecti�es from Different Disciplines. Walter de Gruyter, Berlin,pp. 219–50

Neitz M, Neitz J 2000 Molecular genetics of color vision andcolor vision defects. Archi�es of Ophthalmology 118: 691–700

Rodieck R W 1998 The First Steps in Seeing. Sinauer, Sunder-land, MA

Sharpe L T, Stockman A, Jagle H, Nathans J 1999 Opsin genes,cone photopigments, color vision, and color blindness. In:

Gegenfurtner K, Sharpe L T (eds.) Color Vision: From Genesto Perception. Cambridge University Press, Cambridge, UK,pp. 3–51

Stockman A, Sharpe L T 1999 Cone spectral sensitivities andcolor matching. In: Gegenfurtner K, Sharpe L T (eds.) ColorVision: From Genes to Perception. Cambridge UniversityPress, Cambridge, UK, pp. 53–87

Wandell B A 1995 Foundations of Vision. Sinauer, Sunderland,MA

Wandell B A 1999 Computational neuroimaging: color repre-sentations and processing. In: Gazzaniga M (ed.) The NewCogniti�e Neurosciences, 2nd edn. MIT Press, Cambridge,MA, pp. 291–303

Webster M A 1996 Human colour perception and its adaptation.Network: Computation in Neural Systems 7: 587–634

Wyszecki G, Stiles W S 1982 Color Science—Concepts andMethods. Quantitati�e Data and Formulae, 2nd edn. JohnWiley, New York

D. H. Brainard

Combinatorial Data Analysis

Combinatorial data analysis (CDA) refers to a class ofmethods for the study of relevant data sets in whichthe arrangement of a collection of objects is theabsolutely central concept. Characteristically, CDA isinvolved with either: (a) the identification of arrange-ments that are optimal for a specific representationof a given data set, and where such an exploratoryprocess is typically carried out according to somespecific loss or merit function that guides a combina-torial search over a domain of possible structuresconstructed from the constraints imposed by theparticular representation selected; or (b) a confirma-tory determination as to whether a specific objectarrangement given a priori reflects the observed data,and where such a confirmatory process is typicallyoperationalizedbycomparing theempiricallyobserveddegree of correspondence between some given data setand the specific structure conjectured for it, to areference distribution constructed from the collectionof all possible structures of the same form that couldhave been conjectured.

The boundaries of what CDA might encompass aresomewhat open but generally we would excludemethods based on the postulation of strong stochasticmodels and their specific unknown parametric struc-tures as underlying a given data set. Although CDAmight use or empirically construct various weightingfunctions, the weights so obtained are not to beinterpreted as parameter estimates in some presumedstochastic model viewed in turn as responsible forgenerating the data. Manifest data are emphasizedsolely, and the traditional concern for an assumedrelationship between the data and a restrictivelyparameterized stochastic model is avoided. For

2263

Combinatorial Data Analysis

Copyright # 2001 Elsevier Science Ltd. All rights reserved.International Encyclopedia of the Social & Behavioral Sciences ISBN: 0-08-043076-7

Implicationswww.informedesign.umn.edu

A Newsletter by InformeDesign. A Web site for design and human behavior research.

Seeing ColorColor is the most dominant design ele-ment, and ironically, the most relativeaspect of design. The perception of colorinvolves human physiological and psy-chological responses. Object, light, eye,and brain are involved in a complexprocess of sensation and perception.Color attracts our attention, helps usmake sense of our environment, andaffects our behavior. Color plays a cul-tural role, an informational role, andeven a survival role. It functions on abasic level of appeal and can elicit strongfeelings of like or dislike. Color is asource of sensual pleasure (Pentak &Roth, 2003).

Color Order SystemsWe are familiar with the most commontype of color arrangement—a color wheelarranged in spectral order. Spectralorder is especially pleasing to the humanperceptual system. The spectrum occursin nature in the refraction of light intobands of color—red, orange, yellow,green, blue, and violet. One hue gradatesinto the next, creating a dynamic colorsensation.

Theoretical Color SystemsScientists, artists, and color theoristshave developed variations of the colorwheel. The first wheel appeared in 1611

and was developed by a Finnishastronomer, Aron Sigfrid Forsius andwas soon followed by Newton’s colorwheel in 1704. The primary objectives ofthese systems are to give order to thevariables of color and to concretely rep-resent colors, because “words are incom-plete expression as color” (Munsell,1981). Munsell developed a three-dimensional color tree. The three vari-ables of color — hue, value, and chromaare displayed on plexiglass branches,one for each hue (see Figure 1). Darkervalues of the hue are toward the bottom;lighter values are toward the top.Brighter hues are seen at the outsideperimeter; duller hues are toward thecenter of the tree. A color wheel made ofhats and shoes, featured in an exhibitionin the Goldstein Museum of Design,arranged the objects in spectral order(see Figure 2).

Michel Eugene Chevreul developed asystem to explain how colors affect eachother. As director of the Gobelins tapes-try studio (France), he realized that colorsystems did not account for perceivedcolor and that colors tend to tinge adja-cent hues with its complementary hue.In response, he designed a color circlethat accounted for differences of satura-tion and value within each hue family.He also created a framework about theeffects of simultaneous contrast.

www.informedesign.umn.eduVOL. 03 ISSUE 5

Seeing Color

Typography and Color

Related ResearchSummaries

IN THIS ISSUE

Figure 1: Munsell Color Wheel

Implications

Practical Color SystemsSpecialized color systems are used in product designand manufacturing. Both the Pantone color systemand the Munsell system are widely used. Pantonehas developed color systems and products for thegraphic, interior, textile, architectural, and industri-al design fields. Pantone has also recently begunforecasting color trends in fashion and interiordesign. The primary goal of both the Munsell andPantone systems is to communicate color in a sys-tematic way, leaving little room for error. The CIE(Commission Internationale de l’Eclairage) chro-maticity diagram displays a color matching systembased on light, and it is shaped like a luminositycurve. The system attempts to eliminate differencesof color perception through mechanical measure-ment of the three variables of a color—luminance,hue, and saturation. While these practical color sys-tems help to ensure accurate color specification,color appearance still varies due to lighting, context,and surface quality.

The Effect of Surface Quality on ColorPerceptionSurface quality contributes to the variability of color,“one and the same color evokes innumerable read-ings” (Albers, 1963, p. 1). This variability is due todifferences in the human visual system, light, andthe material and surface quality of the object. Whenwe view the color of an object, we are really seeingreflected light. Objects are typically colored witheither pigment or dyes. Dyes permeate the molecularstructure of the object; pigments lay in a coat of coloron the surface of an object. This difference is evidentin viewing fabric that has been painted versus fabricthat has been dyed.

Surface materiality also affects the appearance of acolor. A smooth, glossy surface will reflect a hue verydifferently than a rough surface, and they tend toreflect more light than a matte or rough surface.Matte or rough surfaces reflect light in a scattered,diffuse manner that randomly mixes the wavelengthsand tends to soften the color, changing it.Transparent materials allow color and light to beseen through them (see Figure 3). Reflection fromglossy paper can make reading a menu or a magazinedifficult just as reflection from a highly polished floorcan create spatial perceptual challenges.

www.informedesign.umn.edu

Where Research Informs Design®

2

Figure 3: Glass designed by Dale Chihuly, Museum of Glass,Tacoma, WA.

Figure 2: A color wheel made from hats and shoes that are in thecollection of the Goldstein Museum of Design.

ImplicationsAlbers (1963) discusses the interdependence of colorwith form and placement, quantity, and quality. It isa constant challenge to predict how a color will lookon the designed object when seen under differentlight sources. While the typical color wheel repre-sents only two or three dimensions, a color systemdeveloped by Albert-Vanel attempted to include vari-ations due to surface quality, light, and human per-ception. This system, called the Planetary color sys-tem and developed in 1983, includes not only hue,value, and chroma, but also accounts for contrastand material.

Dyes and ColorantsThe color of objects is dependent on the pigments ordyes used in the production of the product. Colortrends often evolve out of technological develop-ments. In the mid-1850s, William Henry Perkinsaccidentally developed effective synthetic dyes forwool and silk as he attempted to synthesize quininefrom aniline. He named the color mauve. Otherchemists developed synthetic aniline dyes that weresignificantly brighter and more saturated than earlynatural dyes. This discovery, along with the develop-ment of organic chemistry as a discipline, fueled thedevelopment of numerous synthetic dyes. Neon dyesand pigments that were developed in the mid-1980sresulted in bright fabrics, accessories, and paperproducts.

Color HarmonyThere are strategies for creating color harmony:using similar values or hues, using hues with com-plementary contrast, or using a large number of huesin careful proportions. Constrast provides a sense ofvisual balance. Munsell recommended balancinglight and dark hues, dull and bright hues, and cooland warm hues. A sense of color harmony is basedpartially in human perception and partially in colortrends (see Figure 4).

Human Perception of ColorColor can have a profound effect on humans. It canaffect our brain waves, heart rate, blood pressure,and respiratory rate. Color also affects us emotional-ly. Exposure to color has an effect on our biologicalsystems. Not only does color affect our sense of well-being, but it also may play a role in medical treat-ments for depression, cancer, and bacterial infec-tions.

Visual PerceptionOur perception of color is dependent on light, object,and our eyes and brain. We know that colors areinfluenced by adjacent colors, indeed, it is rare to seean isolated color or color in its pure state. Chevreuldiscussed how colors tend to tinge neighboring hueswith their complement. Including color oppositeswithin close proximity in a particular space can mit-igate this phenomenon. Surgical personnel in hospi-tals wear greenish-blue scrubs to counter-balancethe visual effect of afterimages. During surgery, alleyes focus on the patient and typically see a varietyof pink and red hues. The red receptors in the eyewould become fatigued if not for the color of thescrubs providing the opposite hue and thus balanc-ing the visual experience.

Color contrast is essential for our understanding ofform and legibility. At least a 70% contrast betweenthe background and letterforms is ideal for signs and

www.informedesign.umn.edu

Where Research Informs Design®

3

Figure 4: The colorful facades of Burano, Italy.

Implicationspainted materials. Conversely, too much contrast inan environment may increase anxiety and tension.Sharp contrasts of color on flooring may createuncomfortable illusions for walkers as they deter-mine whether the floor is flat or not. Research hasshown that the most visible combinations of colorsare yellow and black, white and black, white andblue, and red and white.

Psychological Responses to ColorWe all react differently to color. We have differentcolor preferences, and we all have our least favoritecolors. Color response is highly personal. What oneperson believes is a restful color, another may findstimulating. Frequently these color preferences arebased on our own personal experience—a fondly

remembered yellow kitchen that belonged to grand-mother. There are also cultural associations thatinfluence our reactions to color. In several cultures,blue is seen as peaceful, protecting, and soothingcolor. Red typically signifies passion and revolution.There are multiple associations for each color. Forexample, black may be seen as sophisticated or asdepressing. Orange can be warm or aggressive.Yellow can be upbeat or acidic.

Marketing research attempts to discover what colorsinfluence human behavior and how people will actwhen they shop, eat, or travel. Findings by market-ing researchers are typically short-lived, however;trends seem to come and go, and other variables inaddition to color affect behavior. While technologycontributes to color trends, culture and social phe-

nomena also affect the popularity of colors. Fashionprints in the 1960s used the bright palette of colorsknown as psychedelic. These colors were fully satu-rated and were intended to mimic the sensationcaused by drugs (see Figure 5). Most of the informa-tion about color meaning is highly subjective andbased on tacit beliefs, rather than research. There isa significant need for systematic research on colorand human perception.

Typography and ColorTypography, the set of alphabetic characters, numer-als, and symbols used to compose copy, can bemanipulated in any number of ways by a graphicdesigner. Size, typeface, letterspacing, leading (thespace between lines of type), case (upper or lowercase), structure (normal, light, bold, italic, bold ital-ic, etc.), and—of course—color can all be used toimprove the legibility (how easy the text is to read),readability (how inviting the text is to a reader), andthe hierarchy or structure of typeset copy.

While each of the previously mentioned characteris-tics can be manipulated by designers setting type,color is an especially important property. We oftenimagine type (or copy) that is set in black on a whitebackground—this is perhaps the most familiar wayto set type on a printed page. However, when wethink of typography in signage and the built environ-ment, a variety of colors and color combinations,come to mind. Consider the new, colorful green andyellow logo signage of BP (British Petroleum) that isemployed in the design of gas stations. Or, think ofthe familiar white type on a green background ofroad signs. Color is employed frequently in environ-mental signage to create a memorable identity thathelps users navigate a space, remember the businessor company, and create a pleasant impression.

When creating environmental signage, it is critical toconsider some of the variables associated with theapplication of color. Here are a few ideas and tips:

www.informedesign.umn.edu

Where Research Informs Design®

4

Figure 5: Fabric samples from 1960s-era clothing.

Implications www.informedesign.umn.edu

Where Research Informs Design®

5

• Consider the contrast between the color of thetypography and the background to ensure that thetype is easy to decipher and read. Type/back-ground color combinations can cause the text toeither advance or recede (see Figure 6).

• Consider the impact of color on interpretation andunderstanding of the content. What does a redheading indicate versus a brown heading? Doessetting less important information in a brighter,more prominent color impact the order that infor-mation is retrieved?

• Consider the user. Be aware of the cultural contextof the environment and the signage, and considercultural norms for particular colors. For example,in Europe and the US, red typography generallymeans warning or attention. The application ofcolor to type can either play into cultural norms forcolor or can contradict them.

• Consider the lighting levels of the environment.While a color combination may work well whenevaluated in your office, the combination may beinappropriate when the lighting levels are different.

• Consider the properties of the signage material.How will a surface that is reflective or flat changethe legibility of the content? How will lighting levelsinteract with the surface properties?

This is not an exhaustive list of issues to considerwhen applying color to environmental signage andtypography. If possible, it is beneficial to have agraphic designer who understands the interactionsbetween typography, color, and the built environ-ment on a design team when designing environmentswith signage. In addition, InformeDesign hasResearch Summaries about graphic design for thebuilt environment.

References—Albers, J. (1963). Interaction of Color. New Haven,

CT: Yale University Press.—Fehrman, K., & Fehrman, C. (2004). Color: The

Secret Influence. Upper Saddle River, NJ: PrenticeHall.

—Munsell, A. H. (1946). A Color Notation. Baltimore:Macbeth.

—Pentak, S., & Roth, R. (2003). Color Basics.Stamford, CT: Wadsworth.

—Sharpe, D. (1981). The Psychology of Color andDesign. Totowa, NJ: Littlefields, Adams & Co.

—Stromer, K. (Ed.). (1999). Color Systems in Art andScience. Edition Farbe/Regenbogen Verlag.

—Walch, M., & Hope, A. (1990). The ColorCompendium. New York: Van Nostrand Reinhold.

About the Authors: Barbara Martinson, Ph.D.,is the Buckman Professor ofDesign Education in theDepartment of Design,Housing, and Apparel,University of Minnesota.She has taught founda-tions-level color courses for20 years, as well as graphicdesign, design history, andhuman factors courses. Sherecently curated SeeingColor, an exhibition at the Goldstein Museum of

Figure 6: An example of poor and excellent contrast betweentypography and background.

Implications www.informedesign.umn.edu

The MissionThe Mission of InformeDesign is to facilitate designers’

use of current, research-based information as

a decision-making tool in the design process, thereby

integrating research and practice.

Created by: Sponsored by:

6

Design. Her research focuses on design education,learning styles, and the use of digital media in teach-ing. Her favorite color is blue.

Kate Bukoski, author of“Typography and Color,” is aPh.D. candidate in graphicdesign and holds teachingand reasearch assisantshipsin the Department of Design,Housing, and Apparel,University of Minnesota. Her research interests focuson the history and state of the profession of graphicdesign.

Additional Resourceswww.digitalanarchy.com/theory/theory_main.htmlwww.colorsystem.comwww.colormatters.com/colortheory.htmlpoynterextra.org/cp/www.colorcube.com/articles/theory/theory.htmwww.tigercolor.com/ColorLab/Default.htmwww.fadu.uba.ar/sicyt/color/bib.htm http://webexhibits.org/colorart/ch.htmlwww.digitalanarchy.com/theory/theory_main.html

Related Research SummariesInformeDesign has many Research Summaries aboutcolor and related, pertinent topics. This knowledgewill be valuable to you as you consider your nextdesign solution and is worth sharing with yourclients and collaborators.

Bright, Saturated Colors Attract Attention—Color Research and Application

Determining Color in the Built Environment—Color Research and Application

Effects of Office Color Scheme on Workers—Color Research and Application

Color Aids Wayfinding for Young Children—Early Childhood Education Journal

Space and Color Affects Cooperation Among Children—Environment and Behavior

Color Judgment is Influenced by the Aging Eye—Family and Consumer Sciences Research Journal

Light Source, Color, and Visual Contrast—Family and Consumer Sciences Research Journal

Color of Light Affects Psychological Processes—Journal of Environmental Psychology

Color, Meaning, Culture, and Design—Journal of Interior Design

Photos Courtesy of:Barbara Martinson, University of Minnesota (p. 1, 2,4, & 5)Caren Martin, University of Minnesota (p. 3)

© 2002, 2005 by the Regents of the University of Minnesota.

Chapter 13

How does visual memory work?

Photo courtesy of Ann Cantelow. The multichannel neuron model ascribes numbers to channels. The channel numbers store and communicate analog data. They can also be used, in a distinct addressing system, to sequentially query the twigs of visual memory.

Addressing and retrievalFor retrieval, the model requires two types of neurons: 1) an address generating neuron, which drives 2) a data storage neuron. To activate a memory stored as "a thing in a place," a stored datapoint must be addressed at precisely that place. In the specific case of a stored pattern of three bleached disks imported from a photoreceptor, a trio of associated datapoints, twigs, must be addressed, one right after the other.

We have a mechanism for generating sequential addresses. The principle is inherent in the multichannel neuron model. The address generator can be the commutator we have postulated at the axon hillock.

To stimulate the first 9 twigs of memory, #1 through #9, each in turn, requires this sort of circuit. The output lines of the axon driven by the addressing commutator are telodendrions, each corresponding to a channel. In this illustration of this model, telodendrions are numbered in order of their firing. Each individual channel synapses to a dendrite. Each dendrite will be stimulated in its turn, in accordance with the ascending circular order of the addressing commutator.

Each dendrite is a “twig memory”. It stores a channel number that stipulates which channel shall be fired in response to the addressing signal. The effect can be tabulated:

The dendrites, which comprise the twigs of memory in this simple model, are each stimulated in turn. The pattern of bleached disks that each twig has memorized is fired back into the nervous system – precisely replicating the pattern originally dispatched from a single photoreceptor’s outer segment at some time and day in the past. In the table, 9 upticks of the address counter’s commutator correspond to a trio of 3D pixels and 3 frames of a film strip. [A slicker model might use just one address tick to elicit all three datapoints, characterizing intensity, wavelength, phase -- but the point is, visual memory is sampled and read out by the ticking of a sequential address counter. It is probably written in the same way.]

All pixels recorded from the retina at the same time, stored in twigs on other photoreceptor antipodal "trees" will have identically the same time stamp in their address. So simultaneously, synchronously, one pixel from every other “tree” or photoreceptor antipode in the retina of memory is being triggered.

The effect is to pump out of memory a stream of past images -- each image made up of millions of 3D pixels. The system is massively parallel and, therefore, moves whole images all at once. It is lightning fast.

Why don't we see these torrents of images from the past? Why aren't we drowning in images? Because these are not literal images. They are images of the Fourier plane. Fourier images are invisible to us, except perhaps in the special case of LSD users. Literal images may impinge on the consciousness as, in effect, search products, but the search itself is conducted as a Fourier process and is unconscious -- offstage and out of sight.

Numbered synapses -- new evidence, old ideaThe idea there might be some sort of detectable ordering or sequencing of synapses on the dendrites is attributed to Wilfrid Rall, who suggested it in 1964 in support of a wholly different and unrelated model of the nervous system. In the 24 September 2010 issue of Science there is a featured report that reinforces the notion there exists some sort of sequentially ordered input pattern in the dendrites.

In these experiments, a programmed series of successive stimuli is made to “walk” from synapse to synapse along the dendrite. If the stimulus series progresses toward the cell body it is more likely to trigger off action potentials than a programmed series of stimuli that walks the other way, away from the soma, toward the tips of the dendrites.

The front half of this experiment consists of the selective stimulation of a row of individual dendritic spines, one after another, using a laser to precisely localize release of glutamate. The basic technology was outlined here. The back half of the experiment is conventional, and consists of electronic monitoring and tabulation of the axon’s response.

In terms of the multichannel model electrophysiology is difficult to interpret. However, a significant feature of the model is a staircase of firing thresholds. One might speculate that as the stimulus is made to approach the soma, it is finding or ultimately directing a pointer to lower and lower firing thresholds, which is to say, lower channel numbers. These low numbered channels would be more easily triggered than higher numbered channels.

Unfortunately there is easy no way to directly measure or guess the channel number associated with an action potential in passage, if indeed multiple channels exist. Again in terms of the model, a plot of channel numbers versus synapse position on dendrites (or, using different techniques, on the teledendrions) would produce a fascinating picture. In any event it is interesting that even conventional electrophysiology suggests there may be some kind sequential ordering, progression, or directional structuring that underlies a map of dendritic spines.

The modelIn modeling this visual memory system I think it would be best to use automated rotating or looping machinery, just as you would in many familiar recording and playback devices. The rotating machine is the commutator. At each addressing tree, let the loftiest addressing commutators walk forward through time automatically, incrementing higher channel by channel. Rough synchronization among trees should suffice. Now, instead of hardwiring and broadcasting addresses in detail, the retrieval system can simply be given a start date/time and triggered off. A string of retrieval instructions will ensue. The system will, in effect, read itself out like a disk drive.

As a practical matter, the model of a retina of memory should probably be constructed in software. Each tree of memory can be modeled as a disk drive storing analog numbers representing 3D pixels, stacked in serial order, that is, the order or sequence in which they were originally captured from the eye. Millions of disk drives, then, each of relatively modest capacity, comprise a retina of memory. In a primitive animal one would expect to find a single retina of memory. In a sophisticated animal, many.

Let’s say the memory trees pre-exist in a newborn animal and that their twigs are unwritten. Each branch is a point in a commutator sequence, and identifies time (that is, sequence) ranges.

From the point of view of addressing the visual memory, reading and writing are, as in a disk drive, similar processes. The writing commutator walks forward through the present moments, guiding incoming 3D pixels from the eye to a series of novel addresses. To elicit a visual memory a reading commutator, which could be the self-same machine, walks forward through addresses denoting a film strip of past moments.

In effect, the pointer of the base commutator on the address generator, as it ticks ahead, is the pointer of the second hand of a system clock. Although the images are recorded at a stately and regular rate, such as one per second -- the recall can be made to happen as fast as the commutator is made to sweep. And it could scan backwards as well as forwards.

How is a pixel memory deployed?This is an unsolved problem in the model. We have to assume it happens but the answer isn't easy or obvious.

We have stipulated what a 3D pixel memory is: Three numbers -- integers -- that represent a pattern of light recorded from three disks in a single photoreceptor at a particular moment in time. The three numbers are sufficient to specify the instantaneous wavelength, intensity and phase of the incoming light, as read out of a standing wave in the outer segment of the photoreceptor.

We are suggesting these three numbers are configured and stored in the brain as an addressable twig of memory -- three dendritic launch pads for three action potentials to be fired down three specific, numbered axon channels. It is nicely set up, this memory, but how did it happen?

The operation of an initial readout commutator in the addressing neuron seems clear. It simply counts up or down. Other commutators fan out from the initial or system counter. At the upper tier of the addressing tree, the commutators, once toggled, can tick forward “on automatic.”

But what about the commutator in the memory neuron?

In the most basic model of the multichannel neuron, developed in Chapter 2, the neuron is functioning as a sensory transducer. The commutator pointer rotates up to a specific numbered channel in proportion to an input voltage or graded stimulus.

But in the memory neuron, we want the pointer to go, first, straight to a remembered channel. Then, second, to another remembered channel. Then, third, to another remembered channel. Hop hop hop. From the address neuron the memory neuron receives three signals in a sequence, via telodendrions 1, 2, 3. The data neuron fires channels corresponding to three remembered photoreceptor disk positions: 2, 7, 34.

Instead of responding proportionately to an input voltage, as in a sensory neuron, the commutator in the memory neuron is responding discontinuously to a memorized set of three channel firing instructions. So the needle of this commutator must swing, not in response to an analog voltage input, but in response to a pixel memory.

In the multichannel model synapses connect individual channels, rather than individual neurons. It could be that the commutator is simply bypassed, so that the appropriate axon channels are hardwired to the dendritic twigs of memory. Synapses at the soma could suggest a short cut past or a way to overrule the inherent commutator.

Maybe there is some rewiring or cross wiring at the level of the dendritic synapses. To borrow a term of art from the conventional playbook of memory biochemistry, maybe the synapses are subject to "tagging." Maybe biochemical markers delivered into the dendrites when the memory was originally recorded are specifying in some way the channel numbers to be fired.

This model suggests a Y-convergence of three neurons, not just two. One delivers addresses. One stores the

data. A third neuron delivers original data from the retinal photoreceptor – data to be written in sequential order into the dendrites of the memory neuron.

Whatever specific mechanism one might choose or invent, the model requires that pixel memory arriving from a photoreceptor in the eye be stored in an antipodal neuron as a trio or linkage of three distinct channel numbers.

ExperimentOne interesting aspect of this memory model is that it suggests an experiment. We are guessing that the individual channels of an addressing axon are, in effect, split out and made accessible as numbered telodendrions. If there is indeed a numerical succession – a sequential firing order – of the telodendrions, then this should be detectable. We were taught that the telodendrions must fire simultaneously. Is this always true? I bet not.

Superimposed networks Note that we have assumed there exists a double network. Above the information tree there is a second tree, a replica of the first, used to individually address each memory "twig".

The principle of two superimposed networks, one for content and the other for control, is a technical commonplace. An early application was the superimposition of a telegraph network as a control system for the railway network. The egregious present day example is the digital computer, with its superimposed but distinct networks for information storage and addressing.

We are long in the habit of dividing the nervous system into afferent and efferent, sensory and motor, but surely there must be other ways to split it, e.g., into an information network and a addressing network. It is typically biological that one network should be a near replica of the other. Evolution proceeds through replication and modification.

Arborization and addressing capacityThe first anatomist who isolated a big nerve, maybe the sciatic, probably thought it was an integral structure – in essence, one wire. Closer scrutiny revealed that the nerve was a bundle of individual neurons. We are proposing here yet another zoom-down in perspective, this time to the sub-microscopic level . We suspect that each neuron within a nerve bundle is itself a bundle of individual channels.

It follows that the functional wiring of the nervous system is at the level of channels. Synapses connect channels, not neurons. This is why one might count 10,000 synaptic boutons on a single neuron’s soma. The boutons were not put there, absurdly, to “make better contact” nor to follow the textbook model of signal integration. They are specific channel connectors, each with a specific channel number.

The neuroanatomical feature that most interests us at this point is axon branching. This is because branching is of paramount importance in familiar digital technologies for addressing – search trees and other data structures. We have proposed a treelike addressing system for the visual memory in the brain. It is reasonable to ask -- where are the nodes?

Not at the branch points.

Photo courtesy of Ann CantelowBranching in a nerve axon is just a teasing apart and re-routing of the underlying channels. It is not a branching marked by nodes or connections in the sense of an T or Y connected electrical branch, or a logical branch in a binary tree.

For an axon that addresses a dendritic twig of memory, all functional branching occurs at the commutator.

Any anatomical branching downstream of the commutator, such as the sprouting from the axon of telodendrions , simply marks a diverging pathway – an unwinding or unraveling, rather than a distinct node or connection. In other words, the tree is a circular data store. The datapoints are stored at twigs mounted on the periphery of a circle. The twigs are accessible through a circular array of addresses. It is analogous to a disk drive in which the disk holds still and the read-write head rotates.

Photo courtesy of Ann Cantelow

Summary of the technology to this pointThe tree in this photograph is a metaphor for the brain structure which corresponds to, and is antipodal to, a single photoreceptor of the eye. It is one single photoreceptor cell's remote memory warehouse -- a tree of memory.

Each twig is a destination with an address, a neuronal process narrowed down to just two or three channels. For example channels 3, 7 and 29, only, might constitute a given twig. Each twig is a 3D pixel frozen in time. The tree will store as many unique picture elements from the photoreceptor’s past as it has twigs.

As many as 125 million of these trees will constitute a retina of memory. We will look for ways to hack down this number, but for the moment let it stand. The point is, we are talking about millions of trees.

All these trees must be queried simultaneously with a particular numerical address, probably associated with a time of storage, to elicit firing from all the right twigs -- just one twig per tree. Properly addressed, a forest of these trees will recreate, almost instantly, a whole-retina image from memory.

In a primitive animal, it would be sufficient to remember 300 images from the recent past. This could be accomplished with a single addressing neuron, a single commutator. But in a modern mammal, it will be necessary to stack the commutators. A bottom commutator can point to any of 300 other commutators. And each of these can, in turn, point to 300 more commutators. With a simple tree of neurons, which is to say, a logical tree built with commutators, one can very quickly generate an astronomical number of unique addresses. We require one unique address for each twig of the data trees.

Are there enough addresses available in this system to organize a mammalian lifetime of visual memories? Yes. Easily. Are there enough memory neurons to match the addressing capacity of the addressing neurons. Probably not. The neuronal brain that lights up our scanners is probably running its memory neurons as a scratchpad memory. It seems likely there is a deeper store.

But will it work? The memory mechanism we have sketched is probably adequate as a place to start. It would work for a directional eye in which changes in wavelength are highly significant cues to the position and movement of a target. It is a visual memory for retaining the "just now," a film strip comprising a few recent frames.

For an imaging eye, or a human visual memory, this memory system is not yet practical because, in its present form, it is a hog for time and resources.

Bear in mind that this model is extremely fast in comparison with any conventional model of the brain based on single channel all-or-none neurons. Two reasons: 1) It is an analog memory, and 2) it is massively parallel.

But a persistent difficulty with this model is serial recall. It appears this memory has to scroll back through all history to find relevant past images. And each image to be tested for a "hit" is composed from as many as 125 million 3D pixels. This is a huge array to deploy and compare, even though the pixels pop up in parallel. This is why van Heerden's memory seems to be such a dream system -- comparison and recall are instantaneous.

The van Heerden memory has a limitation, however, which a film strip memory does not. If a single face is presented to the van Heerden memory system, it can respond with a class picture in which the face appeared. This supplies context -- a surround of useful information associated in the past with this particular face. However, the system does not automatically position the memory in time.

In a film strip memory, in contrast, progression through time is built in. If the input is an image of a shark, the film strip memory will (like the van Heerden memory) turn up an image that puts the shark in a momentary context from the past. Maybe the shark of memory is freeze-framed in the middle of a school of fish.

But in the film strip memory, the film strip can progress forward through past time. The remembered shark turns, sees you, comes toward you, looms large, opens its mouth. In other words the film strip memory provides not only context and associations -- but also shows cause and effect. Here is a shark. Fast forward. Here are rows of teeth.

It is reasonable to imagine that the memory we have today evolved from a simple (possibly Cambrian) film strip memory of the type we have described. For a simple animal in fixed surroundings, a film strip memory is fairly easy to model and easy to evolve -- to a certain point. But the film strip model soon becomes oppressively slow and heavy with data.

How could this model be speeded up and expanded, that is, modernized? First, by taking full advantage of the Fourier plane. Second, by introducing a metamemory, in effect, a hit parade. Third, by multitasking.

The Fourier FlashlightLet's take a moment to orient ourselves, using the central, red DC spot as a point of reference. The red spot marks, in effect, the center of the Fourier plane. It also marks, probably, the position of the fovea.

We are looking at a structure inside a brain -- the retina's memory -- and the red spot marks that part of the memory antipodal to the foveal cones. The fovea is a wonderful thing but we can ignore it in this discussion. It is the hole in the doughnut in terms of Fourier processing and filtering. In terms of natural history, it can't tell us much. The fovea is a rare and special feature, a splendid particularity of primates, birds, and a few other smart and lucky vertebrates.

But the visual memory evolved in vertebrates that had no fovea. It seems a reasonable guess that the visual memory is grounded on Fourier processing. We have speculated in Chapters 5 and 6 that Fourier processing evolved in vertebrates as means of clarifying a blurry picture of the world obscured by glia, neurons, and vascular tissue because vertebrate photoreceptors are wired from the front.

One could pump a time-series of addresses into the whole forest of memory, but it makes more sense to address the memory very selectively, so that only part of it responds. This approach is indicated in the photo above with a yellow disk -- effectively, a Fourier flashlight.

Recall that any part of the Fourier plane can be transformed into the whole of a literal, spatial image from the world. By selecting such a small part of the whole retinal output, illuminated by the flashlight, we have reduced the data storage and processing problem to a tiny fraction of that associated with the original 125 million neuron source. It is because of this holograph like effect -- the whole contained in each of its parts -- that Karl Lashley was able to physically demolish so much of the visual cortex with lesions without producing a significant loss in the animals' visual memory.Let me emphasize that the Fourier flashlight is a metaphor. It draws a convenient circle around a small population of neurons. In the model this is accomplished by addressing that small population, rather than the whole retina of memory. As the population of neurons and, thus, the flashlight spot gets smaller, the resolution of the literal image that can be recreated by Fourier transformation deteriorates. For rapid scanning and quicker retrieval, one would favor the smallest practical spot. Say a "hit" occurs, that is, a Fourier pattern scanned up from memory is found to match, more or less, a Fourier pattern at play on the retina.

At this point, the spot we are calling the Fourier Flashlight could be expanded in diameter to improve the resolution of the remembered image, broaden the range of spatial frequencies to be included, and perhaps pick up some additional and finer detail.

Where on the retina of memory should we shine the flashlight? Spotlight addressing of the memory map gives us a means to accomplish Fourier filtering. For edge detection, we should select a circle of trees at the outermost rim of the Fourier pattern (and retina), where the highest spatial frequencies are stored. For low spatial frequencies, position a circle near the red DC spot.

For most animals most of the time, an enhancement of high spatial frequencies has significant survival value. Here are two images from the Georgia Tech database, the first literal, the second with high spatial frequencies enhanced.

Remark the sharp definition of the edges of the mirror frame, the clown's arm, and of the edges of the makeup brushes and pencils. In effect, an image in which edges are soft and ill defined has been turned into a cartoon, with heavy outlines emphasizing the edges of objects. This is the information an animal needs immediately. If the animal were looking at a shark in the shadows, filtering for high spatial frequencies would make the shark's shape unmistakable.

Edge enhancement in visual image processing and visual memory has an additional advantage, which is that it creates a very spare, parsimonious image consisting of a few crucial outlines. This important data of high spatial frequency needs to be surfaced quickly for survival purposes.

This yellow band indicates an address map for the outer regions of the Fourier plane, where high spatial frequency information crucial to edge detection is concentrated. These are neurons antipodal to rod cells at the outer periphery of the retina. It is interesting that this neglected outer frontier of the retina might have such a critical survival benefit for the animal. The animal's concept of "an object" arises from edge detection. The uncanny ability to distinguish the integrity of an object even though other objects may intervene is probably rooted in high spatial frequency detection in this part of the retina.

Chopping for speedA memory that is capable of storing a film strip of images is valuable but slow. We can speed it up by massively cropping the images to be scanned for recall, using the Fourier flashlight technique described above. But one must still scan the images accumulated for "all time" to identify the objects currently in focus on the retina. If the object is, in fact, a shark, one doesn’t have time to scan through a lifetime of accumulated memories, in serial order, in order to recognize it.

One solution is to chop the serial film strip into, for instance, one hundred short film strips. Or one thousand. Or ten thousand.

Use these to play multiple Fourier flashlights upon the retina of memory. In this way one could make multiple simultaneous scans and comparisons with the incoming retinal image from the eye.

Maybe the image to be matched is an old automobile. Figuratively, one Fourier flashlight can scan for the cars of the 60s, one for the cars of the 70s, one for cars of the 80s, and one for the cars of the 90s.

This works because each flashlight is addressing a spot in the Fourier plane -- where any and every spot that might be addressed contains all the information needed to match or reconstruct a whole image.

Memory anticipates realityBy multitasking the scans, we are breaking past the cumbersome requirement for serial, linear scanning and recall. It is possible now to see an advantage in importing from the eye a huge retinal Fourier plane. Because of its large area, the incoming Fourier pattern is open and accessible to thousands of simultaneous memory scans.

Think of each Fourier "flashlight" as a projector running, from memory, a short, looped film strip. Looping is easy because the addressing mechanism is a commutator.

Each projected frame in this little movie is a Fourier pattern that corresponds to (and is transformable into) some remembered object.

All these projectors run constantly. In this metaphor, the essential comparator, which is derived from the idea originally conceived by Pieter van Heerden, is a screen. The Fourier flashlights play constantly at spots on one side of the comparator screen. The Fourier plane imported from the eye plays on the other side of the comparator screen. Say the comparator has sensitivity to the sum of the juxtaposed signals on either side of the thin screen. Thus, the comparator will develop a high amplitude signal -- the "Voila!" -- wherever and whenever there is good agreement between a projected pattern from a memory flashlight and an imported pattern from the eye.

There are of course no literal flashlights projecting Fourier patterns onto comparator screens in the brain. The lights and projections are metaphors for processes that are carried out numerically in the model, using channel numbers for addressing, for pixel memory including phase conservation, and for summation.

If this model is viable then the photo above polka dotted with an array of Fourier flashlights is a significant illustration. It explains how we can glance at an object from a bygone époque and immediately identify it. It also explains how that same object can be pictured in different visual contexts captured at several different past

moments.

The visual memory is not an image retrieved by combing through a serial archive of old, static, stored images. The memory is "live," fully deployed in an enormous array, waiting in anticipation for reality to arrive from the retina of the eye.

What does this suggest about the performance of the system? Neuroanatomy has identified, so far, about 30 representations of the retina in the cerebral cortex. By replicating the incoming Fourier plane, one can multiply the area available for the deployment of Fourier flashlights, increasing the number and variety of the arrayed memories that wait in anticipation of an incoming image.

So many flashlights. It has a Darwinian quality. Thousands of memories are on offer all the time. Upon the arrival of a new image on the retina, one or more memorized images shall be selected. It is never necessary to discover an exact fit to the incoming image. The signal from a Van Heerden detector ascends with and reports similarity, so it naturally finds a “best fit.”

The memory of memoriesThe chopped, short, looped film strip memories for images can be further edited or recombined in such a way as to store, in a serial format, only useful images. By this we mean images that have been very frequently matched to incoming images from the retina. Such images are, in effect, played back again and again. This requires a memory for memories. A hit parade of useful associations. We can call such a pared down memory projector a metamemory.

The comparator's criterion for a "hit" is likeness. Similarity. Over time, objects that happen to be alike would steadily accumulate in a metamemory. The notion is roughly analogous to the conventional concept of priming.

A metamemory based on like-kind associations will outscore the serial memories and produce the quickest recalls. In other words, it will succeed and grow. With "likeness" as the selection criterion, one would expect to see the whole system evolve as the animal matures -- growing steadily away from serial recall, which is simply based on recording order, and toward recall based on association and analogy.

For example, a sea animal’s immediate surroundings – a rock, a coral head, a neighbor who is a Grouper, a sprinkled pattern of featherduster worms… all this could be stored as a readily accessible library of constantly recurring rims, edges and patterns and colors. These valuable memory strips could be allocated extra space on the screen, for more detail and higher resolution.

One purpose of this more compact and efficient strip of memory would be to help the animal notice novelty in its immediate environment – anything not familiar gliding into the quotidian scene. A second advantage is a quick read of edible prey and dangerous predators -- and visual elements associated with them -- lines, curves, colors, patterns and textures.

It would be helpful to have a metamemory that excerpts the recent past, the "just now." A metamemory could be written to exclude all the trivial steps that intervene between cause and effect.

A modern creature – a Grand Prix racing driver for example – will have developed a metamemory that strings together the downshift points and apexes of every successive corner and straightaway in Monaco or the Nürburgring.

But note that it is no longer necessary to imagine a stream of memories that arrive and are stored in a linear time sequence like a film strip of a racetrack. An anthropologist might have a metamemory for the totems and icons of every culture she has studied. A biochemistry student will have a metamemory for amino acids, for sugars, for the ox-phos pathway, and branching, non-linear alternative pathways like the phosphogluconate shunt.

One can probably regress this principle, so that there come to be metamemories of metamemories, using key images as tabs, or points of entry. The comparators all operate in the Fourier plane, which is offstage and invisible to the conscious intelligence.

Replacing the retinal image Finally, instead of simply scanning against the incoming image on the retina of the eye, seeking something similar from past experience -- the metamemories might scan against each other’s products, which are images extracted from the past.

An image from a memory associated with a former boyfriend from the 1980s might include a Chinese restaurant from the 1980s. A bright red menu from that restaurant might be matched by a bright red pair of boots that seemed fashionable in 2005.

The requirement for serial recall is now exploded. We can jump from visual fragment to visual fragment, and match memories not just with the eye’s reality of the moment but also with past realities glimpsed and recorded in past moments. The fragments are serial recordings, but they are short, selected, and looped.

ConclusionsThe past and present coexist in the visual memory of the brain.

It seems that a visual memory model structured in this way could mix present and past imagery into something new. In the frequency domain images can be combined and recombined and subsequently Fourier transformed into literal images never before seen. In this way the system can do more than remember. It can invent.

By arraying thousands of Fourier flashlights simultaneously and in parallel, memory recall can be made sudden.

This is a brain model that can actually function using our absurdly slow-moving nerve impulses, which have typical speeds ranging from just 60 mph to 265 mph.

The model is able to do so much work in parallel and simultaneously because of a peculiar property of recordings made in the frequency domain that conserve spatial phase information: Each tiny part one might isolate encodes and can be used to reproduce the whole of a spatial image.

Thousands of elements of an incoming Fourier plane from the retina can be separately and simultaneously compared with past Fourier plane images pumped out of memory. The model is massively parallel, incremental-analog, and massively, simultaneously multitasking.

It requires a multichannel neuron and the conservation of spatial phase. Its playground and operating system is the Fourier plane of the brain’s retina of memory. This crucial Fourier plane of the brain is antipodal to, and is a representation of, the back focal plane of the lens of the human eye.

posted by John Harris at 11:43 AM

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 1

Color Appearance and the Emergence and Evolution of Basic Color Lexicons

Paul Kay Luisa MaffiU. of California, Berkeley Northwestern [email protected] [email protected]

March 7, 1999

AbstractVarious revisions of the Berlin and Kay (1969) model of the evolution of basic

color term systems have been produced in the last thirty years, motivated by bothempirical and theoretical considerations. On the empirical side, new facts about colornaming systems have continually come to light, which have demanded adjustments inthe descriptive model. On the theoretical side, there has been a sustained effort to findmotivation in the vision science literature regarding color appearance for thesynchronic and diachronic constraints observed to govern color terminology systems.The present paper continues the pursuit of both of these goals. A new empiricalquestion is addressed with data from the World Color Survey (WCS) and a revisedmodel is proposed, which both responds to recently raised empirical questions andprovides new motivation from the field of color vision for the observed constraints oncolor naming.

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 2

0 IntroductionThe fact that different languages provide different lexical classifications of color

has long been known. In the nineteenth century, it was not uncommon to infer fromthis observation that languages which fail to make a lexical distinction between whatEuropeans recognize as two qualitatively distinct colors, such as green and blue, do sobecause their speakers cannot discriminate the colors in question perceptually. Forexample, William Gladstone wrote, on the basis of philological investigations ofHomeric Greek, "... that the organ or color and its impressions were but partiallydeveloped among the Greeks of the heroic age" (1858, cited by Berlin and Kay 1969: 135).Similar views were widespread among Gladstone's contemporaries (see Berlin and Kay1969: 134-151). They did not, however, go entirely unchallenged. As early as 1880, theGerman opthalmologist, Hugo Magnus, recognized that a population's failure toimpose a lexical distinction between colors does not necessarily reflect a deficit amongits members in the perceptual ability to discriminate those colors (Magnus 1880: 34-35,discussed in Berlin and Kay 1969: 144ff).

While the nineteenth and early twentieth century students of color vocabulariesworked mostly within the predominantly evolutionary approach to things social andcultural characteristic of the time, with the ascendance in the 1920s, '30s and '40s oflinguistic and cultural relativity, spearheaded by Edward Sapir (e.g., 1921: 219) and B.L.Whorf (e.g., 1956 [1940]: 212 ff.), color came to be singled out as the parade example of alexical domain in which the control of language over perception is patent, that is, of theview diameterically opposed to that of Gladstone and his fellows. Although neitherSapir nor Whorf ever wrote on color words, the presentation of the lexical domain ofcolor as the empirical locus classicus of linguistic relativity and language determinismwas reflected in a small number of highly influential empirical studies (Ray 1952, 1953,Conklin 1955) and in numerous survey and textbook presentations (e.g., Nida 1959: 13,Gleason 1961: 4, Bohannan 1963: 35ff, Krauss 1968).

Berlin and Kay (1969) used a set of stimulus materials developed earlier byLenneberg and Roberts (1956) in a Whorfian-influenced study to assess the meanings ofthe basic color terms of twenty languages and extended their two main conclusions toanother seventy-eight languages reported in the literature. These conclusions were (1)that there are universals in the semantics of color in (probably) all languages: all of themajor color terms they found appeared to be based on one or more of eleven focalcolors, and (2) that there exists an apparent evolutionary sequence for the developmentof color lexicons according to which black and white precede red, red precedes green andyellow, green and yellow precede blue, blue precedes brown and brown precedes purple,pink, orange and gray. While psychologists, including specialists in color vision,largely welcomed these findings (Bornstein 1973a,b, Brown 1976, Collier et al. 1976,Miller and Johnson-Laird 1976, Ratliff 1976, Shepard 1992, Zollinger 1972, 1976, 1979),anthropologists expressed skepticism, principally on methodological grounds (e.g.,Hickerson 1971, Durbin 1972, Collier 1973, Conklin 1973).1

In the ensuing years, a number of empirical studies of color terminology systemsin field settings confirmed the broad outlines of the Berlin and Kay findings, whileamending many details (e.g., Heider 1972a,b, Heider and Olivier 1972, Heinrich 1972,Kuschel and Monberg 1974, Dougherty 1975, 1977, Hage and Hawkes 1975, Berlin and

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 3

Berlin 1975, among many others). These studies led to an early reformulation of theencoding sequence (Berlin and Berlin 1975, Kay 1975). Subsequently, Kay and McDaniel(1978) again reconceptualized the encoding sequence. This reformulation was based on(1) further empirical descriptive work, (2) earlier experiments of Chad K. McDanielworking with William Wooten (McDaniel 1972), which had established the identity ofthe green, yellow and blue Berlin and Kay semantic focal points with the correspondingpsychophysically determined unique hues, and (3) the introduction of a fuzzy setformalism2 (See now Zadeh 1996). The Kay and McDaniel model emphasized (1) thesix primary colors of opponent theory (black, white, red, yellow, green, blue)3, (2) certainfuzzy unions of these categories (notably, green or blue, red or yellow, black or green orblue, white or red or yellow), which are named only in evolutionarily early systems,and (3) the 'binary' colors of the vision literature (e.g., purple, orange), which Kay andMcDaniel referred to as 'derived' categories. These are based on fuzzy intersections ofprimaries and tend strongly to be named only in systems in which all (or most) of theunion- based (or 'composite') categories have already dissolved into their constituentprimaries. Kay and McDaniel also related the universals of color semantics in thismodel, which was based squarely on the six psychophysical primaries of opponenttheory, to the psychophysical and neurophysiological results of R. De Valois and hisassociates (De Valois et al. 1974 [psychophysics of macaque color vision], De Valois et al.1966, De Valois and Jacobs 1968 [neurophysiology of macaque color vision]).

In recent years there have been two additional refinements of the model (Kay,Berlin and Merrifield 1991 [KBM], Kay, Berlin, Maffi and Merrifield 1997 [KBMM]), towhich we will return. Also there have been two major empirical surveys, whoseresults largely support the two broad hypotheses of semantic universals andevolutionary development of basic color term systems. These are the World ColorSurvey, whose results are discussed in this paper and the Mesoamerican Color Survey(MacLaury 1997, and earlier publications cited there).4 Throughout all these revisions,two of the original empirical generalizations of Berlin and Kay (1969) have beenmaintained.

I There exists a small set of perceptual landmarks (that we can now identify withthe Hering primary colors: black, white, red, yellow, green, blue5) whichindividually or in combination form the basis of the denotation of most of themajor color terms of most of the languages of world.6

II Languages are frequently observed to gain basic color terms in a partially fixedorder. Languages are infrequently or never observed to lose basic color terms.7

The various revisions of the 1969 model have been motivated by both empiricaland theoretical considerations. On the empirical side, new facts about color namingsystems have come to light, which have demanded adjustments in the descriptivemodel. On the theoretical side, there has been a sustained effort to find motivation inthe literature on color appearance for the synchronic and diachronic constraintsobserved to govern color terminology systems. The present paper continues thepursuit of both of these goals. A new empirical question is addressed with data fromthe World Color Survey (WCS) and a revised model is proposed, which responds to

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 4

recently raised empirical questions and provides new motivation from the field ofcolor vision for the observed constraints on color naming.

0.1 The Emergence Hypothesis

A tacit assumption made by Berlin and Kay (1969) and maintained throughoutrevisions of the model to date has been the proposition that "all languages possess asmall set of words (or word senses) each of whose significatum is a color concept andwhose significata jointly partition the psychological color space" (Kay in press 1: 1).This assumption has been challenged, explicitly by Maffi (1990a) and Levinson (1997),implicitly by Lyons (1995, in press, cf. Kay in press 2), and by Lucy and the team ofSaunders and van Brakel.8 The rejection of this assumption has been christened theEmergence Hypothesis (EH). According to the EH, not all languages necessarily possessa small set of words or word senses each of whose significatum is a color concept andwhose significata jointly partition the perceptual color space. If we admit the EH as aworking hypothesis, several questions immediately arise.

First, what proportion of the world's languages are non-partition languages, thatis, fail to have lexical sets of simple, salient words whose significata partition theperceptual color space?

Secondly, in the case of partition languages, to what extent and in what mannerdo they conform to generalizations I and II above?

Thirdly, in the case of non-partition languages, to what extent and in whatmanner do they correspond to generalizations I and II?

Regarding the first question, it appears that in the ethnographic present non-partition languages are rare. The data from most languages studied in the WCS give noindication of non-partition status. (The exceptions are discussed in section 3 below.)Also, most reports on color term systems in the literature and in personalcommunications received by the authors give no suggestion that the language beingreported fails to provide a simple lexical partition of the color space. One might objectthat such reports merely betray an unreflecting assumption, based on the reporter'sown language, that every language partitions the color space with a simple lexical set.Such a conjecture is neither provable nor disprovable. In any case, the apparent paucityof non-partition languages in the ethnographic present may not be representative ofhuman history. Specifically, just as there are no two-term ("Stage I" in the model to beintroduced) languages in the WCS sample and very few reported in the literature9, therelative lack of non-partition languages in the ethnographic present may reflect to anunknown degree the (putative) facts that (1) some extant partition languages were non-partition languages in the past and (2) some extinct non-partition languages may haveleft no non-partitioning descendants, or no descendants at all. Again, it is not obvioushow empirical evidence may be brought to bear on such conjectures. We hope that thepresent paper will help stimulate field linguists and linguistic ethnographers toexamine the color lexicons of the languages they encounter for evidence of non-partition status. It is unlikely at this point in world history that many more non-partition languages will be discovered, which makes the discovery and careful study of

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 5

each one all the more important. Philological reconstructions of data on extinctlanguages (e.g., Lyons 1995, in press on Ancient Greek) and exegetical reanalyses ofreports that were originally aimed at different goals (e.g., Lyons in press on Hanunóo,Lucy 1996, 1997 on Hanunóo and Zuni, Wierzbicka, 1996: 306-308 on Hanunóo) areunlikely to cast more than hazy light on the matter. Rather, carefully controlled,contemporary field studies aimed directly at EH issues, like that of Levinson 1997, areneeded. (For discussion, see Kay 1997).

The answer to the second question (How do color-space-partitioning languagessatisfy I and II?) will largely be provided, we hope, by a forthcoming monographreporting the results of the WCS. That monograph will assess in detail the extent towhich each of the 110 languages of the survey fits, or fails to fit, the new modelpresented here.

The present paper also provides an initial attempt to answer the third question(How might non-partition languages satisfy I and II?) by examining the data of Yélîdnye(Levinson 1977) and the relevant data from the WCS. The new model maintains theapplication of generalizations I and II to partition languages embodied in the KBMMmodel, while extending their application to non-partition languages. The goal of thispaper is, therefore, to propose a general model of universals and evolution of basiccolor term systems, which (a) yields a slightly modified version of the KBMM model asthe statistically predominant special case, partition languages, (b) accounts for non-partition (EH) languages and (c) derives these results from independent observationsregarding (i) lexical structure and (ii) color appearance. Additionally, the proposedmodel provides an explanation for the hitherto recalcitrant puzzle posed by theexistence of composite categories comprising both yellow and green (KBM, MacLaury1987, 1997: 74, passim).

1 Principles of the New Model

The model is based on four principles. The first principle derives from linguisticobservations, the other three from observations regarding color appearance.

1.1 Partition

The partition principle subsumes under a broad generalization the specifictendency for languages to provide a small set of basic color terms which jointlypartition the perceptual color space. Studies of other lexical domains by ethnographicsemanticists and structuralist lexicographers have shown a tendency for languages tocontain sets of lexical items which partition certain obvious notional domains, such askin relations, locally observable living organisms, regions of human (and animal)bodies, periods of the solar day, cardinal directions, seasons of the solar year,conversational participants (e.g., as reflected in person/number/gender systems), and soon.10 Ethnographic semanticists have often emphasized the differences in the waysdistinct languages lexically partition a given notionally defined domain. Less oftenthey have called attention to cross-language similarities in the ways certain notionaldomains are lexically partitioned. All such comparisons are based on the tacitassumption that each of the languages being compared partitions the domain lexically.

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 6

This widespread tendency for notionally salient domains to be partitioned by a set oflexemes is what we refer to as the partition principle.

(0) Partition: In notional domains of universal or quasi-universal culturalsalience (kin relations, living things, colors, etc.), languages tend to assignsignificata to lexical items in such a way as to partition the denotata of thedomain.11

The strong tendency of languages to conform to Partition accounts for the rarityof non-parition languages. The fact that Partition expresses a strong tendency, ratherthan an exceptionless rule, is consistent with the fact that non-partition languages doexist.

The amount of information carried by the colors of objects may affect the salienceof the color domain. In a technologically simple society, color is a more predictableproperty of things than in a technologically complex one. Except perhaps for a few pairsof closely related species of birds or of fish, it is rare that naturally occurring objects orthe artifacts of technologically simple societies are distinguishable only by color. Intechnologically complex societies, on the other hand, artifacts are frequently to be toldapart only by color. The limiting case is perhaps color coding, as used in signal lights,electric wires and other color-based semiotic media. But almost every kind of materialthing we encounter in daily life: clothing, books, cars, houses, ... presents us with thepossibility that two tokens of the same type will be distinguishable only, or most easily,by their colors. As the colors of artifacts become increasingly subject to deliberatemanipulation, color becomes an increasingly important dimension for distinguishingthings and hence for distinguishing them in discourse. As technology develops, theincreased importance of color as a distinguishing property of objects appears to be animportant factor in causing languages to add basic color terms, i. e., to refine the lexicalpartition of the color domain (Casson 1997).

The same process provides a plausible reason for the transition from non-partition to partition languages. Specifically, non-partition languages, like early-stagelanguages, may be spoken in societies where color is of relatively low culturalsalience.12 If we assume that cultural salience is promoted by increased functional loadin communication, we expect a rise in technological complexity to both push a non-partition language toward full partition status and cause a language that already has afull partition of the color space to refine that partition, that is, to move further alongthe (partially ordered) universal evolutionary trajectory. On this view, both theevolution of basic color term systems and the evolution toward basic color termsystems result in large measure from increasing technological control of color: astechnological control of color increases, its manipulation in the manufacture ofeveryday artifacts causes it to bear an increasingly greater functional load in everydaylinguistic communication and thereby to achieve greater cultural salience.13 Greatercultural salience of color induces partition of the color space where it does not alreadyexist and leads to increasingly finer partitions of the color space where a partitionalready exists. This process may still be going on (Kay and McDaniel 1978, Chapanis1965).

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 7

1.2 Principles of color term universals and evolution based on color appearance

The three remaining principles of the currently proposed model are color-appearance-based. All presuppose the elemental nature of (1) the four primary huesensations of opponent theory: red, yellow, green and blue and (2) the two fundamentalachromatic sensations black and white. The overwhelming majority of visionscientists interested in color appearance and categorization now accept the basic natureof these six color sensations on the basis of a wide range of psychophysical andcognitive psychological evidence.14 The model of Kay and McDaniel (1978) mistakenlyequated these six primary color sensations with the six classes of cells identified by DeValois et al. (1966) in the parvocellular layer of the macaque lateral geniculate nucleus(LGN), and called them fundamental neural response categories.15 These six cell typescannot simply constitute the neural substrate of the six primary color sensationsbecause, among other reasons, (1) they contain nothing corresponding to the shortwavelength red response and (2) the points at which the spectrally opponent cells areneither excited nor inhibited are not in the right places to produce the observed uniquehue points (Derrington et al. 1984, Abramov and Gordon 1994, Abramov 1997). Weshould note, however, that it is psychophysical experiments that have established theshort wavelength red response and the unique hue points in a variety of ways,involving diverse techniques such as hue cancellation and hue scaling (Boynton andGordon 1965, Jameson and Hurvich 1955, Hurvich and Jameson 1955, Ingling et al.1995, Sternheim and Boynton 1966, Werner and Wooten 1979, Wooten and Miller 1997.See Hardin 1988, Chapter I for general discussion.) The elemental character of black,white, red, yellow, green and blue in human color sensation, within a conceptualframework that includes the notions of chromacy/achromacy, unique hues andopponent processes, is no longer thought to be grounded in macaque LGN neurons, butthis framework is nonetheless broadly accepted by vision scientists as the best way toorganize a wide range of psychophysical, cognitive-psychological and animal-behavioral observations (Abramov 1997, Abramov and Gordon 1997, Bornstein 1997,Hardin 1988, Ingling 1997, Kaiser and Boynton 1996, Miller 1997a,b, Sandell et al. 1979,Shepard 1994, Van Laar 1997, Werner and Bieber 1997, Wooten and Miller 1997, Sivik1997.16 For dissent, from two distinct points of view, see Jameson and D'Andrade 1997and Saunders and van Brakel 1997).

1.2.1 Black and White

The first principle governing the refinement of lexical partitions of the colorspace is given by the fact that object recognition is possible without color, e.g., in blackand white movies and photographs. In fact, it is often claimed – probably anexaggeration, according to Wooten and Miller (1997) – that the rods are only active inscotopic (low illumination, black and white) vision and contribute nothing to photopic(bright illumination, color) vision. Certainly, the cones transmit luminance as well aschromatic information (De Valois and De Valois 1975, 1993). It is clear nonetheless thatobjects can be distinguished rather well at levels of illumination too low to stimulatethe cones to give rise to hue sensations. The distinction between spectral sensitivity(spectral opponency) and spectral non-sensitivity (spectral non-opponency) is alsoreflected in the anatomical and physiological distinction between the magna layer andparvo layer cells of the lateral geniculate nucleus. "The great majority, if not all, of the

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 8

P-cells in a macaque... have responses that are spectrally opponent ... while M-cells aregenerally spectrally non-opponent..." (Abramov 1997: 101, citing the primary literaturefor both observations.) Macaque color vision has been shown to be in essential respectslike that of humans by De Valois et al. (1974). At a more phenomenal level we canobserve that people with no color vision (those suffering from achromatopsia) oftenhave no problem with object recognition (Davidoff 1997, Mollon 1989). In short, wehave a black-and-white vision system that gives us most of shape discrimination andobject recognition with color vision laid on top of it. Indeed, students of vision haveoccasionally been led to speculate how and why our species should have evolved colorvision at all (e.g., Mollon 1989, Hardin 1992). A person lacking color vision is not blind.A person lacking the black-and-white vision necessary to recognize objects is blind.

The partitioning principle motivated by these observations is:

(1) Black and White (Bk&W): Distinguish black and white.

1.2.2 Warm and CoolA distinction between "warm" and "cool" colors has long been recognized by

color specialists from both the arts (e.g., art critics and historians and teachers ofpainting) and the sciences. Red, yellow and intermediate orange are "warm"; green andblue are "cool." Hardin (1988: 129ff) provides an excellent discussion of bothexperimental and philosophical considerations of the warm/cool distinctions,beginning with Hume and concluding, in part, "These explanations [of the warm/coolhue associations and cross-modal associations] are of varying degrees of persuasiveness,but they should at least caution us not to put too much weight on any single analogicalformulation. However, they should not blind us to the striking fact that there is aremarkable clustering of oppositions which correlate with this hue division" (Hardin1988: 129). Early experiments (e.g., Newhall 1941) established red as a warm hue. Morerecent experiments (Katra and Wooten 1995), controlled for brightness and saturation,have shown that English-speaking subjects' judgments of warm color peak in theorange region and cover reds and yellows, while judgments of cool color peak in theblue region and cover greens and blues. Judgments of warmth/coolness also correlatewith saturation (saturated colors are judged warm), but not significantly with lightness.These groupings of basic hue sensations into warm and cool agree with those commonin the art world. A recent study of color term acquisition in two-year-olds, besidesfinding surprising control of color terms in very young children, found no significantdifferences among colors in the age at which they were acquired but did find that "therewas some evidence that our subjects maintained the warm-cool boundary; in generalthey make more within- than across-boundary errors" (Shatz et al. 1996: 197). Bothartistic tradition and recent experimental evidence thus point to an affinity between redand yellow on the one hand and between green and blue on the other. A recent colormodel based on observed cone frequencies (De Valois and De Valois 1993, 1996) positsan intermediate stage of chromatic information processing that consists of twochannels: one red/yellow and one green/blue (See Kay and Berlin 1997 for discussion ofthe possible relevance of this model to cross-language color naming). Thepsychological color space, so-called, is notoriously lacking in a reliable long-distancemetric17. We take the facts mentioned in this paragraph to indicate, albeit indirectly,

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 9

that red and yellow are experienced as in some respect similar and that green and blueare experienced as similar in that same respect.

The partitioning principle motivated by the warm and cool groupings of hues is:

(2) Warm and Cool (Wa&C): Distinguish the warm primaries (red and yellow) fromthe cool primaries (green and blue).

1.2.3 Red

The final principle we propose for explaining the ways languages go aboutlexically partitioning the color space is the apparent salience of red among the huesensations. Despite the intuitive judgment, shared by vision specialists and lay people,that red is somehow the most salient of hues, non-anecdotal support for this idea is notoverwhelming. Humphrey (1976) writes

I shall list briefly some of the particular evidence which demonstrates how, in avariety of contexts, red seems to have a very special significance for man. (1)Large fields of red light induce physiological symptoms of emotional arousal –changes in heart rate, skin resistance and the electrical activity of the brain. (2) Inpatients suffering from certain pathological disorders, for instance cerebellarpalsy, these physiological effects become exaggerated – in cerebellar patients redlight may cause intolerable distress, exacerbating the disorders of posture andmovement, lowering pain thresholds and causing a general disruption ofthought and skilled behaviour. (3) When the affective value of colours ismeasured by a technique, the 'semantic differential', which is far subtler than asimple preference test, men rate red as a 'heavy', 'powerful', 'active', 'hot'colour. (4) When the 'apparent weight' of colours is measured directly by askingmen to find the balance point between two discs of colour, red is consistentlyjudged to be the heaviest. (5) In the evolution of languages, red is withoutexception the first colour word to enter the vocabulary – in a study of ninety-six[sic, actually ninety-eight] languages Berlin and Kay (1969) found thirty [sic,actually twenty-one] in which the only colour word (apart from black and white)was red. (6) In the development of a child's language red again usually comesfirst, and when adults are asked simply to reel off colour words as fast as they canthey show a very strong tendency to start with red. (7) When colour vision isimpaired by central brain lesions, red vision is most resistant to loss and quickestto recover (Humphrey 1976: 97f).

It is disquieting to note, however, that the only reference provided for thevarious claims in the passage just cited is to Berlin and Kay (1969) and that both of thenumbers reported from that work are inaccurate.

Following the publication of Berlin and Kay (1969), Floyd Ratliff, a distinguishedvision scientist, attempted to provide motivation from color science for the 1969 model(Ratliff 1976). Among the elements he sought to explain was the prominence of red.Ratliff noted that the long-wavelength cones are very frequent in the fovea and aremuch more sensitive in the long wave end of the spectrum than the other two cone

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 10

types. This line of argument has not, to our knowledge, been found persuasive. Forexample, Wooten and Miller (1997: 86) point out that Ratliff established no linkbetween the observation of a dense population of long-wavelength sensitive cones inthe fovea and the subjective salience of red. They note further that subjective colorsensations are linked quite indirectly to cone responses, probably at cortical levelsbeyond the primary visual area.

At this time, the firmest warrant we can find for the apparent prominence of redamong the hue sensations comes from research on color term acquisition. There havebeen several studies of the acquisition of color terms in English-speaking children.Some of these have noted a weak correlation of the order of acquisition of basic colorterms with the original Berlin and Kay 'encoding' sequence and others no suchcorrelation. An observation that has not previously been made about these studies andother studies of acquisition of color terms by English-speaking children is that in everycase in which acquisition data is reported by term, red is the first of the hue termsacquired (Wolfe 1890 [data reproduced in Descoeudres 1946: 119]; Winch 1910: 475,passim; Heider 1971: 453. Table 3; Johnson 1977: 309f, Tables 1, 3 and 4). The same fact –that red is the first hue term acquired by children – is also evidenced by studies onGerman (Winch 1910: 477); Spanish (Harkness 1973: 185, Figure 4); Russian (Istomina1963: 42f, Tables 6, 7); Italian (Winch 1910: 456-457); French (Descoeudres 1946: 118f),Mam [Mayan] (Harkness 1973: 184, Figure 3 [red and green tied for first for 7-8 yearolds]); Setswana [Bantu] (Davies et al. 1994: 701-702, tables 4 and 5 [Setswana termsonly]); and West Futuna (Dougherty 1975, table 5.718). In every study we have found inwhich a difference between colors was reported in the order with which childrenacquire terms for them, the term for red was the first hue term acquired.19 The finalprinciple of color naming expresses the primacy of red among the hue sensations.

(3) Red: Distinguish red.

2. The WCS Data to be Accounted For

The 110 basic color terminology systems of the WCS were classified by KBMM (p.33, Figure 2.4) into eleven basic types, based on the combinations of Hering primaryterms they contain. As shown in Figure 1, Stages I (two terms) and II (three terms) eachcorrespond to a single type, Stage III (four terms) comprises three types, Stage IV (fiveterms) three types, and Stage V a single type. (Two stages hypothesized by KBMM,IIIBk/Bu and IIIY/G, have been eliminated from the model because no instances of themhave been discovered in the WCS data.20) In Figure 1, columns represent evolutionarystages, every stage containing one more basic color term than the preceding stage.KBMM recognized languages in transition between types. In Figure 1, an arrowindicates the transitions from the type occurring on its left to the type toward which itpoints. For example, Stage II systems can develop into either type IIIG/Bu or typeIIIBk/G/Bu.21 Stage IIIBk/G/Bu systems can develop into systems of either Stage IVG/Bu orStage IVBk/Bu, and so on.

Progression through successive stages, starting with a two-term systems andadding a term at each stage, results from the interaction of the Partition principle withthe six Hering primaries. Initially, minimal application of Partition dictates division of

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 11

the color space into two categories. Of course, Partition alone doesn't tell us what thesecategories will be, that is, how the primaries will be grouped in the cells of the resultingpartition. That is the job of the three additional, color-appearance-based principles.Each of the three remaining principles is applied in order until an unequivocal result isdetermined. At each succeeding change point, this process is repeated: Partition isapplied, minimally, to dictate that the number of cells (= named basic color categories =basic color terms) be increased by one. Then principles (1), (2) and (3) are applied inorder until an uniquivocal result regarding the nature of the new partition is achieved.(Whenever application of a principle is decisive in determining the refinement of thepartition, principles of lower priority are not consulted. Eventually there remains onlyone possible refinement of the existing partition, so application of principle (0) sufficesto produce an unequivocal result and no other principles are consulted.)

The order of application (1) > (2) > (3) expresses an empirical hypothesisregarding the relative importance of the principles. This order seems to correlate –impressionistically speaking – with the wieght of the evidence we have been able toamass for principles (1), (2), and (3) in sections 1.2.1, 1.2.2, and 1.2.3, respectively. Theordering of Parition (0) before the other three principles follows from the fact that whatwe are using the principles for is to refine a partition and principle (0) is the one thatsays, "Refine the partition."

WRYBk/G/Bu (IIIBk/G/Bu)

➙ ➘

WRYGBk/Bu (IVBk/Bu)

➘

W/R/Y

Bk/G/Bu

➙

W

R/YBk/G/Bu

➚➙

WR/YG/BuBk (IIIG/Bu)

➙

WRYG/BuBk (IVG/Bu)

➙

WRYGBuBk

WRY/G/BuBk (IIIY/G/Bu)

➚ ➙

WRY/GBuBk (IVY/G)

➚

I II III IV VFigure 1. Types and Evolutionary Stages of Basic Color Term Systems

(Adapted from KBMM, Figure 2.4, page 33)

2.1 The Main Line of Basic Color Term Evolution

The languages of the WCS indicate five possible paths ending in Stage V, whichcan be traced by following the arrows from stage to stage in Figure 1. These define fiveevolutionary trajectories, identified as A, B, C, D, E, in Table 1.

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 12

A: I ➙ II ➙ IIIG/Bu ➙ IVG/Bu ➙ VB: I ➙ II ➙ IIIBk/G/Bu ➙ IVG/Bu ➙ VC: I ➙ II ➙ IIIBk/G/Bu ➙ IVBk/Bu ➙ VD: ?22 ? IIIY/G/Bu ➙ IVG/Bu ➙ VE: ? ? IIIY/G/Bu ➙ IVY/G ➙ V

Table 1. Five Evolutionary Trajectories of Basic Color Term Systems

The evolutionary trajectories of Table 1 are not equally frequent in the WCS data.A single trajectory, which we call the main line of color term evolution, accounts forthe vast majority of WCS languages. Ninety-one of the 110 WCS languages (83%)belong either to one of the five stages of Trajectory A or to a transition between two ofthese stages, as shown in Figure 2, where an outlined numeral within bracketsrepresents the number of WCS languages found at the corresponding stage and anoutlined numeral between brackets represents the number of WCS languages found intransition between the stages indicated.23

W/R/Y Bk/G/Bu

➙

WR/Y 6Bk/G/Bu

3 ➙

WR/YG/Bu Bk 3 (IIIG/Bu)

4 ➙

WRYG/Bu 41Bk (IVG/Bu)

11 ➙

WRYG 2 3BuBk

I II III IV VFigure 2. Main Line (Trajectory A) of Evolutionary Development of Basic Color

Lexicons. Total number of languages represented is 91 (83% of WCS languages)24.

2.2 Accounting for the Main Line of Color Term Evolution

Our internal representation of color, independent of language, appears to play animportant role in determining the evolution of color term systems. Our task in thepresent section is to explain why Stage I systems have the particular shape they do andwhy each type of basic color lexicon on the main line (Figure 2) evolves into thesucceeding type. The evolutionary sequence of the main line can be motivated byassuming, as we have above, that at each stage transition principles (0) Partition, (1)Bk & W, (2) Wa & C and (3) Red operate in that order until an uniquivocal result isreached. We assume that Partition acts minimally and incrementally. That is, webegin with the color space lexically partitioned into just two cells, that is, namedcategories, each cell (named category) representing a union of some subset of the sixfuzzy sets corresponding to the primary colors and then at each new stage, reapplicationof Partition and the other three principles adds a single new cell (i.e., term), until the sixprimaries have each received a distinct basic color term.

2.2.1 Stage I

Stage I is motivated as follows. Principle (1) [ Bk&W] dictates that one cell of thetwo-cell partition shall contain B and the other W. Principle (2) [Wa&C] dictates thatone cell shall contain both R and Y and the other shall contain both G and Bu. Itremains to be determined whether the warm primaries will be grouped with W andthe cool with Bk or vice versa. Yellow is an inherently light color. Perusal of the

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 13

systematically arranged stimuli of any standard color order system, e.g., Munsell, NCS,or OSA, shows that low lightness colors of the same dominant wavelengths as yelloware not seen as yellow, but as orange, olive, brown, or something hard to name. To saythat Y is an inherently light color it to say that Y and W have an inherent affinity. Thefact that one of the warm colors, Y, is seen as similar to W correlates with, and partiallyexplains, the apparently universal association of the warm hues with W and, therefore,of the cool hues with Bk in Stage I systems.25

Independent of the inherent lightness of Y, in discussing various cross-modalassociations to the warm/cool distinction in hues, Hardin (1988: 129) notes that amongthese are active/passive, exciting/inhibiting, up/down, and positive/negative (in anon-evaluative sense). Hardin advances – cautiously – the speculation that we mayhave sensitivity to the polarity of opponent processes, in particular that we may havesome neural level which records such facts as that R, Y and W each represent excitationof their opponent process, while G, Bu and Bk represent inhibition of thecorresponding opponent mechanisms (1988: 130). Our interest here is not to evaluateHardin's speculation regarding a possible neural basis for the white/warm, dark/cooland correlative cross-modal associations but simply to note the existence of thewhite/warm and dark/cool associations.

The strength of the association of warm hues with W and of cool hues with Bk isreinforced by experiments performed by James Boster (1986). In one experiment Bostergave twenty-one naive English-speaking subjects eight color chips, representing focalexamples of the categories black, white, red, orange, yellow, green, blue and purple. Theinitial instruction was to sort the chips into two groups "on the basis of which colorsyou think are most similar to each other..." (Boster 1986: 64). The overwhelmingpreference was to put white, red, orange and yellow into one group and green, blue andblack and purple into the other. Two thirds of Boster's subjects chose this exactdivision into two subsets. (There are 2,080 ways a set of eight elements can be dividedinto two non-empty subsets.) In a second experiment, the same instruction was givento a group of eighteen subjects, using as stimuli the eight color words rather than thecolored chips. Substantially the same result was obtained.

2.2.2 From Stage I to Stage II

As indicated above, in deriving each stage from the preceding stage, we apply tothe earlier system principles (0), (1), (2) and (3) in that order. Applying Principle (1) to aStage I system means that either W and R/Y are given separate terms or that Bk andG/Bu are given separate terms. Principle (2) is irrelevant to the decision whether R/Yor G/Bu gets a separate term, so principle (3) is consulted. Principle (3) is relevant,dictating that the division be made between W and R/Y, since this choice promotes thedistinguishing of R more than if the division were made between Bk and G/Bu. Theresult is a Stage II system, with terms for W, R/Y, and Bk/G/Bu.

2.2.3 From Stage II to Stage IIIG/Bu

Applying Principle (1) to a Stage II system requires the extraction of Bk fromBk/G/Bu, since W already has a separate term. The result is a Stage IIIG/Bu system, with

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 14

terms for W, Bk, R/Y and G/Bu. Principles (2) and (3) have no opportunity to applybecause application of (1) has been sufficient to add a term, satisfying Partition.

2.2.4 From Stage IIIG/Bu to Stage IVG/Bu

Principle (1) does not apply to a Stage IIIG/Bu system, since Bk and W already haveseparate terms. Principle (2) is uninformative with respect to breaking up R/Y or G/Bu.Principle (3) requires breaking up R/Y into R and Y. The result is a Stage IVG/Bu system,with terms for Bk, W, R, Y and G/Bu.

2.2.5 From Stage IVG/Bu to Stage V

Since a Stage IVG/Bu system contains only one composite category, G/Bu,application of Partition alone is sufficient to determine the result. To satisfy Partition,G/Bu must be divided into G and Bu, yielding a Stage V system with terms for Bk, W,R, Y, G, and Bu. Partition, Bk&W, Wa&C and Red, operating in that order, account forthe evolution of eighty-three percent of the WCS languages.

2.3 Less Frequent Evolutionary Trajectories

As shown in Figure 1, there are also cases of WCS languages in which thetransition from Stage II to Stage III involves separating R and Y, instead of Bk andG/Bu. The result is a Stage IIIBk/G/Bu system. Such systems are involved inevolutionary trajectories B and C in Table 1. A Stage IIIBk/G/Bu system can in turndevelop into either a Stage IVBk/Bu or a Stage IVG/Bu system, as shown in Figure 3.Figure 3 adds these types, and related transitions, to the main line of developmentshown in Figure 2.

WRYBk/G/Bu IIIBk/G/Bu

1➙

1➘

WRYG 3Bk/Bu IVBk/Bu

1➘

W/R/Y Bk/G/Bu

➙

WR/Y 6Bk/G/Bu

1➚

3➘

WRYG 23BuBk

WR/YG/Bu 3Bk IIIG/Bu

4 ➙

WRY 41G/BuBk IVG/Bu

11➚

I II III IV V Figure 3, Evolutionary Trajectories A, B and C26

As shown in Figure 3 and note 26, an additional ten languages (10% of the WCStotal) reflect the minority choice of splitting R and Y in going from Stages II to III, rather

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 15

than dividing Bk/G/Bu into Bk and G/Bu. This amounts to promoting Principle (3)[Red] over Principles (1) [Bk&W] and (2) [Wa&C]. Of these ten languages, one is intransition from a mainline type (II) to a non-mainline type (IIIBk/G/Bu), while five are intransition from a non-mainline type to a mainline type.27

Summarizing to this point, 101 of the 110 WCS languages (92%) showexceptionless operation of Partition – that is, no evidence of the EH – either in theirpresent condition or, by plausible inference, in a former state. Of these, ninety-one(90%) conform to the ordering of Partition and the three color-appearance-basedprinciples, Bk & W, Wa & C, and Red: (0) > (1) > (2) > (3). Ten of these 101 languages(10%) order Principle (3) over Principles (1) and (2) at some point in their evolutionarydevelopment. We turn our attention now to the exceptional cases, the languages inwhich Partition appears to fail at least partially, and in which the EH consequently findssupport.28

3 Predictions of the Model for Non-Partition (EH) Languages

The only thoroughly documented non-partition language of which we are awareis not a WCS language but Yélîdnye, a Non-Austronesian language of Rossel Island(Papua New Guinea), reported in Levinson (1997). Because Levinson undertook hisinvestigation of Yélîdnye color naming with the EH specifically in mind and because hecollected, in addition to the WCS color naming tasks, a fuller range of morphosyntacticand usage information than it was possible to ask the WCS field linguists to record, hisreport of a positive finding on the EH deserves close attention. In very brief summary,Yélîdnye has basic color terms for B, W and R and a secondary but well establishedsimple term for a certain red color, specifically that of a shell used in traditional inter-island (Kula) trade.

The three basic terms kpêdekpêde 'black', kpaapîkpaapî 'white' and mtyemtye(or taataa) 'red' are recognizable as reduplications of nominal roots denoting a treespecies, a pure white cockatoo and a "startling crimson" parrot, respectively. Levinsonnotes that there is a "regular", that is, partially productive, derivational pattern in thislanguage according to which reduplication of a nominal root may derive an adjectivedenoting a salient property of the denotatum of the noun. For example, mty:aamty:aa'sweet' < mty:aa 'honey'. Levinson points out that if one knows the white cockatooand red parrot one might well guess the meanings of the reduplicated forms of theirrespective names to mean 'white' and 'red', though of course one could not be certainthat some other salient property (such as the loud screech of the parrot) was not beingpicked out. One might wish to argue on the basis of these observations that the red andwhite words of Yélîdnye fail the first criterion of basicness of Berlin and Kay: "... [the]meaning [of the color word] is not predictable from the meaning of its parts" (1969: 6).Having raised the issue, and suggesting that it may be one that arises in manylanguages of Oceania and Australia, Levinson appears convinced in the end that thewhite and red terms of Yélîdnye should be considered basic color terms, whatever anarrow application to them of the Berlin and Kay criteria might yield. But he suggeststhat observations such as these might be interpreted as casting doubt on the claim thatYélîdnye has, aside from kpêdekpêde 'black', any basic color terms in the sense of Berlin

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 16

and Kay (1969) and perhaps that some languages of Oceania or Australia have any basiccolor terms at all.

On closer examination, this fear appears to be groundless. Yélîdnye kpaapîkpaapî'white' and mtyemtye (or taataa) 'red' do not fail the Berlin and Kay (1969: 6) criterionof non-predictability of meaning. At issue is the proper understanding of(non)-predictability of meaning. Makkai (1972) makes a relevant distinction between'encoding idioms' and 'decoding idioms' (see also Fillmore, Kay and O'Connor 1988:540f). An expression that a speaker would not know how to assemble from knowledgeof everything else in a language is an encoding idiom. An expression that a hearerwould not be able to interpret from knowledge of everything else in a language is adecoding idiom. There are many encoding idioms which are not decoding idioms, thatis, there are many expressions which are interpretable on first hearing but that onewouldn't know how to form from knowledge of everything else in the grammar. Forexample, on first hearing one of the expressions light as a feather, heavy as lead orquick as a wink, any English speaker could probably figure out exactly what was meant,but one could not know in advance that these are conventional ways of saying 'verylight', 'very heavy', 'very quick', even knowing that English contains a pattern [A as aN] for forming expressions meaning 'very A'. There is no way to know in advance thatone may say, for example, light as a feather, easy as pie or easy as duck soup, but not*light as an ash, *easy as cake or *easy as goose fritters, or that one may say one (two, ...)at a time, but not *one at the time [as in French], *one to a time, *one by the time, etc.,without learning each separate fact.

Analogously, Yélîdnye could have reduplicated forms of the word meaning leaffor 'green', of turmeric or banana for 'yellow', and of sky for 'blue', but it doesn't.29

Even though this particular derivational process of Yélîdnye is used frequently (and isin that sense "regular"), the speaker of Yélîdnye nonetheless has to memorizeseparately each of the cases in which it is used, so each of these cases represents aseparate encoding idiom although it is possible that none are decoding idioms. If weinterpret the non-predictability criterion for basic color terms as requiring that suchterms be encoding idioms – which seems appropriate since language users have tospeak their language as well as understand it – then kpaapîkpaapî and mtyemtye (ortaataa) meet the non-predictability criterion for basicness – as they meet all the otherB&K criteria. Insofar as similar reduplication process are reflected in the color terms ofother Oceanic and Australian languages, as Levinson suggests, the same argumentapplies to them.

The Bk, W and R terms of Yélîdnye are not extended; this is not a Stage IIlanguage, in which, for example, the term that includes Bk also includes G and Bu andthe term that includes R also includes Y and orange. Interestingly, there are fixedphrasal expressions denoting each of the colors G, Y and Bu. The most highlyconventionalized and widely shared of these is for G, then Y, then Bu – the last subjectto a large number of phrasal expressions and considerable interspeaker variation. TheBk and W terms are somewhat more firmly established and subject to less interspeakervariation than the basic R terms (due perhaps to dialect synonymy in R, plus possibleinterference from the Kula-shell term). Much of the color space is simply unnamed byany expression Levinson was able to elicit.

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 17

Yélîdnye seems clearly to be a non-partition language, i.e., one testifying to thecorrectness of the EH. On the other hand, Yélîdnye has a very Berlin and Kay (1969)'feel' to it: the best established terms are for Bk and W, then R, all basic, then non-basicG, Y, and Bu in that order, and after these nothing worth mentioning. Yélîdnye is not apartition language. It nevertheless exhibits the salience of Bk and W dictated byPrinciple (1 ) and the salience of R dictated by Principle (3). Principle (2) has no scope tooperate in Yélîdnye, since in this non-partition language there are no composite termsfor Principle (2) to apply to.

3.1 WCS Evidence for the EH

Levinson suggests strongly that for Yélîdnye we should think of Bk, W and R asreceiving basic color terms (the last with two competing synonyms, deriving fromdifferent dialect names for the eponymous parrot), and these only. There are alsoseveral languages in the WCS with well-established words for BK, W and R (notextended), with varying ways of treating lexically the rest of the colors. We mustcaution here that the WCS data were not collected specifically to test the EH and that welack for these data much information on the morpho-syntactic status of the terms andthe kind of ethnographic observation of their use in natural discourse that would bevery useful for assessing the applicability to these languages of the EH. Nevertheless,some patterns may be observed.

The existence of languages with basic terms only for (non-extended) Bk, W and Ris consistent with the fact that Bk, W and R are singled out by Principles (1) and (3),while Y, G, and Bu are not distinguished per se by any principle of the model. Suchlanguages are spoken in communities in which color as such may not have achievedsufficient cultural salience, and thus functional load in communication, for Partition totake full effect in the color domain, leaving the field open, as it were, for Principles (1)and (3) to cause only the inherently most prominent color sensations to receive simplenames. So far our model has yielded an explanation for color systems with basic termsfor Bk, W and R only, and which therefor do not partition the perceptual color space. Ifa language has gone this far and no farther, we will find well established terms for Bk,W and R and widespread variability on WCS tasks in the rest of the color space, withmany competing terms and little agreement among speakers. To a significant degree,six of the seven of the WCS languages that remain to be discussed fit this descriptionand the seventh, Cree, although a partition language in the present, may be inferred tohave been non-partition in the reconstructable past..

3.2 The Residue Predicted: Y/G/Bu Terms

Suppose a language has developed non-extended terms for B, W, and R, ignoringPartition. If Partition now asserts itself, a composite term for Y/G/Bu appears,producing a Stage IIIY/G/Bu system. This type of system contains basic terms for Bk, Wand R and a composite term covering Y, G and Bu. The WCS sample contains two clearexample of such systems, Karajá (Brazil) and Lele (Chad).

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 18

Systems of this type are reported elsewhere in the literature. For example,Arrernte (Pama-Nyungan, Australia) apparently had such a system (David Wilkins, pc1998, see also Spencer and Gillin 1927, noted in Berlin and Kay 1969: 67f).30 Kinkade(1988) reconstructs a Proto-Salishan Y/G/Bu term because of clear etymologicalrelatedness of terms including or restricted to Y and terms including or restricted to Buin contemporary Salishan languages. (See also the discussion in MacLaury 1997: 74,passim) The notion that the IIIY/G/Bu systems developed historically from systems likeYélîdnye – with basic color terms for Bk, W, and R and no lexical partition of the colorspace – is of course speculative. We have no historical record or detailedreconstruction of such a development for either WCS or non-WCS languages. But thisconjecture fits the model to the available data very neatly, accounting for evolutionarytrajectories D and E of Table 1. The questions signaled by the question marks in Table 1have now been addressed.

3.2.1 The Yellow/Green Mystery Resolved

The development of a Y/G/Bu term as a delayed assertion of Partition provides aplausible explanation of the puzzle regarding the origin of Y/G terms.31 In the Kay andMcDaniel model, every language is assumed to start out as a Stage I (fully partitioned)system and to develop further via successive division of composites until all sixlandmark colors receive separate terms. Since Y and G belong to distinct composites atStage I, it is a mystery under this model how Y/G composites ever come into being(See KBM for further discussion.) Under the present model, which allows for the EHand therefore does not assume that all languages start from a fully partitioned Stage Isystem, a plausible scenario for the genesis of Y/G composites suggests itself. Once asystem with restricted Bk, W, and R plus a composite Y/G/Bu exists (Stage IIIY/G/Bu), itmay develop further in either to two ways. If the Y/G/Bu composite splits into Y andG/Bu the result is a mainline Stage IVG/Bu system, with terms for Bk, W, R, Y andG/Bu (Trajectory D). But if the other possible split of the Y/G/Bu category occurs, intoY/G and Bu, the result is a Stage IVY/G system, with terms for Bk, W, R, Y/G, and Bu(Trajectory E). Among WCS languages, Cree is an example (the sole example) of such asystem and it is the only WCS language with a Y/G composite. To our knowledge, allother languages reported to contain Y/G composites are also of this type, Stage IVY/G.The developmental scenario just sketched, in which Y/G/Bu categories result from thelate imposition of Partition on Bk-W-R (only) languages and in which Y/G compositesresult from the breakup of Y/G/Bu composites, eliminates from the theory the logicallypossible but unattested KBMM Stage IIIY/G type, with terms for W, R, Y/G and Bk/Bu.MacLaury (1987) has documented Y/G terms in several Salishan languages, confirmingthe earlier reports of Kinkade and others. Kinkade (1988) and MacLaury (1997: 74,passim) conclude that some G/Bu, Bu and Y/G terms observed in modern Salishanlanguages reflect a Proto-Salishan Y/G/Bu term.

To summarize the Y/G story: Y/G/Bu terms arise when ascendency of the color-appearance-based priciples (1) and (3) over Partition and (2) leads to the naming of Bk,W and R, leaving the rest of the color space unnamed; then Partition exerts itself,resulting in the creation of a Y/G/Bu term to name the rest of the primary colors andpartition the space. The inherently unstable Y/G/Bu category (containing the opponentcolors Y and Bu) usually breaks down into Y and G/Bu, leaving no trace of its prior

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 19

existence (since the resulting mainline Stage IVG/Bu type more usually arises from thebreakup of R/Y in a mainline IIIG/Bu system). But occasionally Y/G/Bu breaks downinto Y/G and Bu, producing a Stage IVY/G system, with terms for Bk, W, R, Y/G and Bu.

3.2.2 Mopping Up: Four EH Languages?

Finally, the WCS files include four languages which appear to represent mixedcases of the patterns outlined above, in the sense of including terms clearly centered onBk, W, and R, with two or more conflicting patterns competing for the remaining area.The single generalization that brings these cases together is that the regions of the colorspace corresponding to Bk, W and R are well named (including either a separate nameor inclusion in a standard composite category like Bk/G/Bu), while the strategy fornaming the remaining areas is some combination of (1) extension of the Bk, W, Rterms according to the usual story of composites, (2) existence of a special Y/G/Bu (orNot-[Bk/W/R]) term, or (3) relatively strong secondary terms for Y, G, Bu, or G/Bu (orsome subset thereof, akin to the Yélîdnye pattern). These languages tend also to bethose in which there is unusual interspeaker variation in the use of shared terms and amarked degree of idiosyncrasy in the selection of terms used.

Culina (Peru, Brazil) is similar to Karajá and Lele in containing terms for W, Rand an extended yellow term that covers much of G and Bu, especially in the lightershades. There is, however, no Bk term, but instead an unmistakable Bk/G/Bu term.Mundu (Sudan) represents a similar situation. There are clear terms for W, R, andBk/G/Bu, but there is also a highly salient term which includes Y, G and Bu, issomewhat focused in Y, and which seems to gloss best as 'everything which is notblack, white or red'. Moreover, Mundu contains a secondary term largely synonymouswith the one just mentioned but much less well established. Culina and Mundu bothseem to mix the W, R, Y, Bk/G/Bu strategy (Stage IIIBk/G/Bu) with the W, R, Bk, Y/G/Bustrategy (Stage IIIY/G/Bu).

The final two languages, Kuku-Yalanji and Murrinh-Patha (both Australian)illustrate most clearly the pattern of Bk, W, R plus confusion. In this respect they comethe closest in the WCS sample to the Yélîdnye pattern in which only restricted Bk, Wand R receive basic color terms. Kuku-Yalanji has well-established terms for Bk, W,and R, although the Bk term shows some extension into Bu (as well as into Br, which iscommon). The R term, ngala-ngala (< ngala 'blood) does not include yellow. Thelanguage contains two additional major terms, although these are less well establishedthan the first three. One, of these, kayal, is used regularly by only half of the speakersconsulted, maps as a G/Bu term for the language as a whole, is focused in G, anddenotes only G for some speakers. It also means 'unripe' according to the WCS fieldlinguists, H. and R. Hershberger. Oates (1992: 126) gives kayal with the gloss '[color]green' only, indicating that the word is among those "not recognised by speakers today"[Recall that the WCS data were gathered fourteen years before the Oates dictionary wasproduced.] Oates also contains an entry kalki 'unripe'. Only nine of twenty WCScollaborators used kayal with a well-established green or grue sense; kayal is not a basiccolor term of Kuku-Yalanji. There is also a word used by seventeen of the twentyKuku-Yalanji speakers for everything outside of Bk, W and R proper, burrkul (orburkul). However, it is clear that collaborators with well-established words for green or

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 20

grue do not use burrkul for those colors. The Hershbergers gloss burrkul as 'non-descript', 'dirty' and 'anything which is not black, white or red'. The last gloss seemsaimed less at the conceptual content of the word than at the way it is deployed in theWCS naming task. Oates lists burkul, not among the color words, but among"Describing Words Relating to Things", giving its gloss as 'not clear, not clean, murkyor dirty, said about water, windows, mirrors, photos, skin' (Oates 1992: 83). burrkul isnot a basic color term of Kuku-Yalanji.

Murrinh-Patha presents perhaps the most confusing array of terms in the WCS.In addition to standard Bk, W, and R terms (with the Bk term thipmam extended a bitinto Bu, as well as into Br, and the R term bukmantharr not extended into Y), there arefour other widely used terms: ngatin (used by twenty-one of the twenty-five WCScollaborators), wudanil (twenty-four speakers), tumamka/tupmanka (nineteenspeakers) and wipmanarri (fifteen speakers). ngatin appears in the pooled data to be aY/G term, but it is used by some speakers for yellow/orange/(brown) only, by someothers for G/Bu only, and by some for G only. wudanil is used by one or anotherspeaker for virtually everything outside of Bk, W, and R. Its distribution on the WCStasks lead one to infer that it might be a non-color term, like Kuku-Yalanji burrkul('non-descript, not clear, not clean,...') and could be used for any surface appearance forwhich the speaker does not have an apt descriptor. However, Michael Walsh (pc 1998)is unable to corroborate that gloss. "[wudanil] could be a verb form which as beenconventionalized to refer to colours but could also have an independent (verbal) life ofits own." tumamka/tupmanka appears to be a widely extend, low consensus G/Buterm if one considers the aggregate mapping, but there is great interspeaker variation inhow the term is used. For some speakers tumamka/tupmanka is blue, for some G/Bu,for many nothing so easy to describe. Walsh writes (pc 1998) that tumamka/tupmankaalso appears to be a verbal form. Finally, wipmanarri covers approximately the samerange of colors as Warlpiri walyawalya (< walya 'earth'), which can denote deepbrowns, reddish browns, lighter – yellowish– browns and oranges, yellowish salmons,pinkish purples and other light purples. This is just about the range of colors earthtakes on in the central Australian desert, where Warlpiri is located (although we don'thave comparable information for the area in which Murrinh-Patha is spoken).However, there is no indication in Walsh's information that wipmanarri has anetymological relation to earth, possibly being related instead to the body-part word for'back'. The Murrinh-Patha Bk, W and R terms are much better established than the lastfour discussed (and some less frequent terms that we haven't discussed here).Murrinh-Patha fits the best of any language in the WCS sample the formula Bk, W, Rplus confusion32.

4 Summary

This paper presents a model of color term evolution employing one language-based principle, Partition, and three color-appearance-based principles: Bk&W, Wa&Cand Red. The Emergence Hypothesis is defined as the possibility that not all languagesobey Partition perfectly in the color domain. Straightforward application of these fourprinciples, with the ranking: Partition > Bk&W > Wa&C > Red, defines the main lineof color term evolution (Trajectory A of Table 1, Figure 2), accounting for 91 (83%) ofthe languages in the WCS sample. When Red supersedes Bk&W and Wa&C at the

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 21

transition from Stage II to Stage III, the possibility of two additional types is created,accounting for an additional 10 WCS languages, bringing the part of the total WCSsample accounted for to 101 (92%) (Figure 3, Trajectories B and C.) Two more languagesdepart non-wildly from any of the nine types in Figure 1 but do not challenge the EH(See note 22), bringing the number of non-EH languages to 103 (94% of the WCS total).The remaining seven languages show, to varying degrees, evidence for the possibleoperation of the EH. Two of these, Karajá and Lele are IIIY/G/Bu languages, illustratingTrajectory D. One language, Cree, illustrates Stage IVY/G (Trajectory E). The remainingfour languages (Culina, Mundu, Kuku-Yalanji and Murrinh-Patha) all show Bk, W,and R prominence, with a mixture of other strategies, combined with considerableinterspeaker variability.

A plausible solution to the apparent mystery of Y/G composites is provided bythe current model: EH languages may develop somewhat along the lines of Yélîdnye,assigning basic terms, according to principles (1) [B&W] and (3) [Red], only to restrictedBk, W, and R, violating Partition. Subsequently, Partition comes into play and aY/G/Bu term appears, covering the remaining primary colors.33 (There is somesuggestive evidence that Y is the most common focus for this term, but the data are sosparse that no reliable conclusion can be drawn here.) In some cases, the Y/G/Bu termmay then divide into Bu and Y/G terms. According to Kinkade (1988) and MacLaury(1997: 74, passim) this appears to have happened in some Salishan languages.34

Since the original Berlin and Kay (1969) study, there have been numerous fieldstudies by linguists and anthropologists which have added data to test and refine thetheory of universals and evolutionary development of basic color term systems. Tothis we can add the Mesoamerican Color Survey and the WCS. This line of researchhas resulted in several reformulations of the evolutionary model and will probablycontinue to do so. Recently, a striking aspect of this tradition of research has consistedin the complex of observations and speculations we have referred to globally as theEmergence Hypothesis. The reformulations of the evolutionary model have, since1978, also been guided by an effort to explain whatever universals in color semantics wecan by independent findings from the vision literature. It is encouraging that thepresent reformulation of the model (1) covers a wider range of partitioning languagesthan any model hitherto, (2) is based more more firmly on independent principlesgoverning color appearance than previous models, (3) sheds some new light on non-partitioning languages and on what the relation of these may be to the partitioninglanguages, their evolutionary sequence, and the color appearance factors that appear tounderly it and (4) goes some way toward solving the hitherto unresolved problem ofcomposite (fuzzy union) categories comprising both yellow and green.

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 22

ReferencesAbramov, I.

1997 Physiological mechanisms of color vision. In Hardin and Maffi (1997) 89-117. [See below]

Abramov, I. and J. Gordon1994 Color appearance: On seeing red – or yellow, or green, or blue. AnnualReview of Psychology 45: 451-485.

Abramov, I. and J. Gordon1997 Constraining color categories: The problem of the baby and the bathwater.Behavioral and Brain Sciences. 20 (2): 179-180.

Berlin, Brent1992 Ethnobiological classification : principles of categorization of plants andanimals in traditional societies. Princeton, N.J. : Princeton University Press.

Berlin, Brent and Elois Ann Berlin 1975 Aguaruna color categories. American Ethnologist 2: 61-87.

Berlin Brent and Paul Kay1969 Basic Color Terms: Their Universality and Evolution. Berkeley and LosAngeles: University of California.

Bohannon, Paul1963 Social Anthropology. New York: Holt, Rinehart and Winston.

Bornstein, Marc H.1973a The psychophysiological component of cultural difference in color namingand illusion susceptibility. Behavioral Science Notes 1: 41-101.1973b Color vision and color naming: A psychophysiological hypothesis ofcultural difference. Psychological Bulletin 80: 257-285. 1997 Selective vision. Behavioral and Brain Sciences 20 (2): 180-181.

Boster, James1986 Can individuals recapitulate the evolutionary development of colorlexicons? Ethnology 26 (1): 61-74.

Boynton, R.M. and J. Gordon1965 Bezold-Brücke shift measured by color-naming technique. Journal of theOptical Society of America 55: 78-86.

Brown, Roger W.1976 Reference. Cognition 4: 125-153.

Casson, Ronald W.1997 Color shift: evolution of English color terms from brightness to hue. InHardin and Maffi (1997) 224-239.

Chapanis, Alphonse1965 Color names for color space. American Scientist 53: 327-346.

Collier, George A.1973 Review of Basic Color Terms. Language 49: 245-2481976 Further evidence for universal color categories. Language 52: 884-890.

Conklin, Harold C.1955 Hanunóo color categories. Southwestern Journal of Anthropology 11: 339-344.1973 Color categorization: Review of Basic Color Terms, by Brent Berlin and PaulKay. Language 75: 931-942.

Davidoff, Jules

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 23

1997 The neuropsychology of color. In Hardin and Maffi (1997) 118-134.Davies, Ian, Greville Corbett, Harry McGurk and David Jerrett

1994 A developmental study of the acquisition of colour terms in Setswana.Journal of Child Language 21: 693-712.

Derrington, A.M, J. Krauskopf and P. Lennie (1984)Chromatic mechanisms in lateral geniculate nucleus of macaque. Journal ofPhysiology 357: 241-265.

Descoeudres, Alice1946 Le développement de l'enfant de deux à sept ans. Neuchatel et Paris:Delachaux & Niestlé.

De Valois, Russell L., I. Abramov and G.H. Jacobs1966 Analysis of response patterns of LGN cells. Journal of the Optical Society ofAmerica 56: 966-977.

De Valois, Russell L. and G.H. Jacobs1968 Primate color vision. Science 162: 533-540.

De Valois, Russel L. and Karen K. De Valois1975 Neural coding of color. In E.C. Carterette and M.P. Friedman (eds)Handbook of Perception. Vol V. New York: Academic. 117-166.1993 A multi-stage color model. Vision Research. 33: 1053-1065.

` 1996 A three-stage color model – comment. Vision Research. 36:833-836.De Valois, Russell L., Herman C. Morgan, Martha C. Polson, William R. Mead, and

Elaine M. Hull1974 Psychophysical studies of monkey vision–I. Macaque luminosity and colorvision tests. Vision Research 14: 53-67.

Dougherty, Janet D.1975 A universalist analysis of variation and change in color semantics. BerkeleyPh. D. dissertation.1977 Color categorization in West Futunese: Variability and change. InSociocultural Dimensions of Language Change. B.G. Blount and M. Sanches,eds., New Yor/London: Plenum. pp. 133-148.

Durbin, Marshall1972 Review of Basic Color Terms. Semiotica 6: 257-278.

Fillmore, Charles J., Paul Kay and M.C. O'Connor1988 Regularity and idiomaticity in grammatical constructions: the case of letalone. Language 64 (3) 501-538.

Gladstone, William E.1858 Studies on Homer and the Homeric Age. London: Oxford University Press.

Gleason, H.A.1961 An Introduction to Descriptive Linguistics. New York: Holt, Rinehart andWinston.

Hage, Per and Christen Hawkes1975 Binumarien color categories. Ethnology 24: 287-300.

Hård, A. and L. Sivik1981 NCS-Natural Color System: a Swedish standard for color notation. ColorResearch and Application 6 (3): 129-138.

Hardin, C.L.1988 Color for Philosophers: Unweaving the Rainbow. Indianapolis, IN: Hackett.

Hardin, C.L.

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 24

1992 The virtues of illusion. Philosophical Studies 67: 153-166.Hardin, C.L and Luisa Maffi (eds).

1997. Color Categories in Thought and Language. Cambridge, UK: CambridgeUniversity Press.

Harkness, Sara1973 Universal aspects of learning color codes: A study in two cultures. Ethos1(2): 175-200.

Heider, Eleanor Rosch1971 "Focal" color areas and the development of color names. DevelopmentalPsychology 4 (3): 447-455.1972a Universals in color naming and memory. Journal of ExperimentalPsychology 93: 1-20.1972b Probabilities, sampling and the ethnographic method: The case of Danicolour names. Man 7: 448-466.

Heider, Eleanor Rosch and Donald C. Olivier1972 The structure of the color space for naming and memory in two languages.Cognitive Psychology 3: 337-354.

Heinrich, Albert C.1972 A non-European system of color classification. Anthropological Linguistics14: 220-227.

Hering, Ewald

1964 [1920] Outlines of a theory of the light sense. Translated by L. M. Hurvichand D. Jameson. Cambridge, Mass: Harvard.

Hickerson, Nancy1971 Review of Berlin and Kay (1969). International Journal of AmericanLinguistics 37: 257-270.

Humphrey, N.K.1976 The colour currency of nature. In T. Porter and B. Mikelides (eds) Color forArchitecture. New York: Van Nostrand.

Hurvich, L.M. and D. Jameson 1955 Some qualitative effects of an opponent-colors theory: II. Brightness,saturation and hue in normal and dichromatic vision. Journal of the OpticalSociety of America 45: 602-616.

Ingling, C.R., Jr. 1997 Constraints on the definitions of "unique hues" and "opponent channels".Behavioral and Brain Sciences. 20 (2): 194-195.

Ingling, C.R., Jr., Jonathan P. Barley, and Nadeem Ghani(1995) Chromatic content of spectral lights. Vision Research 36 (16): 2537-2551.

Istomina, Z. M.1963 Perception and naming of color in early childhood. Soviet Psychology andPsychiatry 1 (2): 37-45.

Jameson, D. and L.M. Hurvich1955 Some quantitative aspects of an opponent-colors theory: I. Chromaticresponses and spectral saturation. Journal of the Optical Society of America 45:546-552.

Jameson, K. and R.G. D’Andrade

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 25

1977 It’s not really red, green, yellow, blue: An inquiry into perceptual colorspace. In Hardin and Maffi (1977) 295-319.

Johnson, E. G.1977 The development of color knowledge in preschool children. ChildDevelopment 48: 308-311.

Katra, E. and B.R. Wooten1995 Perceived lightness/darkness and warmth/coolness in chromaticexperience. ms.

Kaiser, Peter K. and Robert M. Boynton1996 Human Color Vision, 2nd edition. Washington, D.C.: Optical Society ofAmerica

Kay, Paul1975 Synchronic variability and diachronic change in basic color terms. Languagein Society 4: 257-270.In press 1 The emergence of basic color lexicons hypothesis: A comment on"The vocabulary of colour with particular reference to Ancient Greek andClassical Latin" by John Lyons. The Language of Colour in the Mediterranean,Alexander Borg (ed) Wiesbaden: Otto Harrassowitz.In press 2 Sur la méthodologie de l'étude comparative des termes de couleur.Anthropologie et Sociétés. [English version available atwww.icsi.berkeley.edu/~kay/color.meth.ps]

Kay, Paul and Brent Berlin1997 There are non-trivial constraints on color categorization. Behavioral andBrain Sciences 20: 196-202.

Kay, Paul, Brent Berlin, Luisa Maffi and William Merrifield1997 Color naming across languages. In Color Categories in Thought andLanguage. C.L. Hardin and Luisa Maffi, eds., Cambridge, England: CambridgeUniversity Press.

Kay, Paul, Brent Berlin and William Merrifield1991 Biocultural implications of systems of color naming. Journal of LinguisticAnthropology 1: 12-25.

Kay, Paul and Chad K. McDaniel1978 The linguistic significance of the meanings of basic color terms. Language54: 610-646.

Kinkade, M. Dale1988 Proto-Salishan color. In W. Shipley (ed) In Honor of Mary Haas. Berlin:Mouton de Gruyter.

Krauss, Robert M.1968 Language as a symbolic process. American Scientist 56: 265-278.

Kuschel, Rolf and Torben Monberg1974 'We don't talk much about colour here': A study of colour semantics onBellona Island. Man 9: 213-242.

Lenneberg, Erik H. and John M. Roberts1956 The Language of Experience: A study in Methodology. Memoir 13 ofInternational Journal of American Linguistics.

Levinson, Stephen A.

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 26

1997. The theory of basic color terms and Yélîdnye. Working papers of theCognitive Anthropology Group, Max Plank Institute for Psycholinguistics.Nijmegen.

Lucy, John A.1996 The scope of linguistic relativity: An analysis and review of empiricalresearch. In J. Gumperz and S. Levinson (eds) Rethinking Linguistic Relativity.Cambridge University Press. 37-69.1997 The linguistics of color. In Hardin and Maffi (1997) 320-346.

Lyons , John A.1995 Colour in language. In T. Lamb and J. Bourriau (eds) Colour: Art andScience. Cambridge University Press.In Press The vocabulary of colour with particular reference to Ancient Greek andClassical Latin. In The Language of Colour in the Mediterranean, Alexander Borg(ed) Wiesbaden: Otto Harrassowitz.

MacLaury, Robert.E.1987 Color-category evolution and Shuswap yellow-with-green. AmericanAnthropologist 89: 107-124.(1997) Color and Cognition in Mesoamerica. Austin, Texas: University of TexasPress.

Maffi, Luisa1988a World Color Survey report. Department of Anthropology, U.C., Berkeley.ms. 33 pp.1988b World Color Survey typology. Department of Anthropology, U.C.,Berkeley. ms. 29 pp.1990a Cognitive anthroplogy and human categorization research. Department ofAnthropology, U.C., Berkeley. ms. 113 pp.1990b Somali color term evolution: Grammatical and semantic evidence.Anthropological Linguistics 32: 316–334.

Magnus, Hugo1880 Die geschichtliche Entwickelung des Farbensinnes. Leipzig: Viet.

Makkai, Adam1972 Idiom Stucture in English. The Hague: Mouton.

McDaniel, Chad K.1972 Hue Perception and Hue Naming. Unpublished B.A. thesis. HarvardUniversity.

Miller, David L.1997a Beyond the elements: investigations of hue. In Hardin and Maffi eds (1997)151-162.1997b Over the rainbow: The classification of unique hues. Behavioral and BrainSciences 20 (2): 204-205.

Miller, George A. and P. Johnson-Laird1976 Language and Perception. Cambridge, Massachusetts: Harvard UniversityPress.

Mollon, J.D.1989 "Tho' she kneel'd in that place where they grew ...": The use and origins ofprimate color vision. Journal of Experimental Biology 146: 21-38.

Newhall, S.M.1941 Warmth and coolness of colors. Psychological Record 4: 198-212.

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 27

Nida, Eugene A.1959 Principles of translation as exemplified by Bible translating. In OnTranslation. Reuben A. Brower, ed., Cambridge, Massachusetts: HarvardUniversity Press.

Oates, Lynette F.1992 Kuku-Yalanji Dictionary. Alsbury, NSW: Graeme van Brummelen.

Palmer, Stephen A.in press Color, Consciousness and the isomorphism constrain. Behavioral andBrain Sciences.

Pokorny, Julius,1948-69 Indogermanisches etymologisches Worterbuch. Bern: A. Francke .

Ratliff, Floyd1976 On the psychophysiological bases of universal color terms. Proceedings ofthe American Philosophical Society 120: 311-330.

Ray, Verne1952 Techniques and problems in the study of human color perception.Southwestern Journal of Anthropology 8: 251-959. 1953 Human color perception and behavioral response. Transactions of theNew York Academy of Sciences (series 2) 16: 98-104.

Sapir, Edward1921 Language. New York: Harcourt, Brace.

Sandell, J.H., Gross, C.G., and Bornstein, M.H.1979 Color categories in macaques. Journal of Comparative and PhysiologicalPsychology 93: 626-635.

Saunders, B.A.C and J. van Brakel1988 Re-evaluating basic colour terms. Cultural Dynamics 1: 359-378.1995 Translating the World Color Survey. In Geuijen, K., D. Raven and J. deWolf (eds) Post-Modernism and Anthropology. Assen, Netherlands: vanCorcum.1997 Are there non-trivial constraints on colour categorization? Behavioral andBrain Sciences 20: 167-228.

Shatz , Marilyn, Douglas Behrend, Susan A. Gelman, and Karen S. Ebeling1996 Colour term knowledge in two-year-olds: evidence for early competence.Journal of Child Language 23: 177-199.

Sivik, Lars1997 Color systems for cognitive research. In Hardin and Maffi (1997: 163-193).

Shepard, Roger1992 The perceptual organization of colors. In The Adapted Mind. J. Barkow, L.Cosmides and J. Tooby, eds., Oxford: Oxford University Press.1994 Perceptual-cognitive universals as reflections of the world. PsychonomicBulletin and Review 1 (1): 2-28.

Spencer, B. and F.K. Gillin1927 The Arunta: A Study of a Stone Age People. London: Macmillan.

Sternheim, C.E. and R.M. Boynton1966 Uniqueness of perceived hues investigated with a continuous judgmentaltechnique. Journal of Experimenal Psychology 72: 770-776.

Taylor, John R. and Robert E. MacLaury, eds

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 28

1995 Language and the cognitive construal of the world. (Trends in linguistics.Studies and monographs 82). Berlin, New York: Mouton de Gruyter

Van Laar, D.1997 Ekphrasis in colour categorisation: Time for research, or time forrevolution? Behavioral and Brain Sciences. 20 (2): 210.

Werner, J.S. and M.L. Bieber1977 Hue opponency: A constraint on colour categorization known fromexperience and experiment. Behavioral and Brain Sciences. 20 (2): 210.

Werner, J.S. and B.R. Wooten1979 Opponent chromatic mechanisms: Relation to photopigments and huenaming. Journal of the Optical Society of America 69: 422-434.

Wierzbicka, Anna1996 Semantics: Primes and Universals. Oxford: Oxford University Press.

Whorf, Benjamin L.1956 [1940] Science and Linguistics. In Language, Thought and Reality: TheCollected Papers of Benjamin Lee Whorf, John B. Carroll, ed., Cambridge,Massachusetts: MIT Press. Originally published in Technology Review 42: 229-231, 247-248.

Winch, W.H.1910 Color-names of English school-children. American Journal of Psychology 21(3): 453-482.

Wolfe, H.K.1890 On the color-vocabulary of children. Nebraska University Studies 1 (3) July1890.

Wooten, B.R. and D. Miller1997 The psychophysics of color. In Hardin and Maffi (1997) 59-88.

Zadeh, Lotfi A.1996 Fuzzy sets, fuzzy logic, and fuzzy systems: selected papers by Lotfi A. Zadeh(Editors: George J. Klir & Bo Yuan). Singapore, River Edge, N.J.: World Scientific.

Zollinger, Heinrich1972 Human color vision: An interdisciplinary research problem. Palette 40: 1-7.1976 A linguistic approach to the cognition of colour vision. Folia Linguistica 9:265-293.1979 Correlations between the neurobiology of colour vision and thepsycholinguistics of colour naming. Experientia 35 :1-8.

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 29

ACKNOWLEDGMENTSBrent Berlin provided much appreciated comments on an earlier draft. He andWilliam Merrifield have been valued colleagues throughout the World Color Survey.We also acknowledge with gratitude the advice of David Nash regarding theinterpretation of the WCS Warlpiri, Kuku-Yalanji and Murrinh-Patha data, DavidWilkins regarding the interpretation of his Arrernte data, as well as the WCS Warlpiri,Kuku-Yalanji, and Murrinh-Patha materials, and Michael Walsh regarding theinterpretation of the WCS Murrinh-Patha data. David Wilkins also allowed us toreview original data on Arrernte color naming, focal choices, and discourse use,collected by him in 1997. Our heartfelt thanks to all these colleagues. Errors are thecontribution of the authors.

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 30

NOTES

1It is perhaps worthy of passing note that the qualms regarding experimental method were expressedalmost exclusively by non-experimentalist anthropologists, while interested psychologists, all of whomwere experimentalists, apparently accepted the rough-and-ready experimental procedures of Berlin andKay because of the robustness of their results (See, for example, Boynton 1997:135f). Collier, both ananthropologist and an experimentalist, is a special case. In (1973) he expresssed the suspicion that theBerlin and Kay results might be an artifact of their stimuli providing maximum available saturation ateach hue/lightness coordinate. Subsequently, Collier et al. (1976) reported an experiment in which thishypothesis was examined and rejected, confirming the Berlin and Kay results at a non-maximal, uniformlevel of saturation.2 Fuzzy sets allow for degrees of membership. For example a yellowish orange color can be thought of as,say, 25% red and 75% yellow, that is a member of the fuzzy set red to the degree .25 and of the fuzzy setyellow to the degree .75. The membership of an individual x in the union if two fuzzy sets, A, B, is themaximum of its membership in either. The membership of an individual in the intersection of two fuzzysets is the minimum of its membership in either. For a non-technical introduction to the basics of fuzzy settheory, see Kay and McDaniel (1978); for full technical detail see Zadeh (1996).3 Contemporary color vision theory recognizes the six primary colors, originally posited in the opponenttheory of Ewald Hering (1964 [1920]), as arranged in three opponent pairs: black/white, red/green,yellow/blue. Any color percept can be formed by combining two or more of these colors perceptually (not aspigments). Red, yellow, green and blue are the unique hues. That is, these four hues and only these can beseen as unmixed. Orange is seen as a mixture of red and yellow, chartreusse is seen as a mixture of yellowand green, but yellow, although it falls between orange and chartreuse on the hue circle, is not seen as amixture of orange and chartruese. Along with black and white, the four unique hues provide the primarylandmarks, or cardinal points, of perceptual color space, with other colors located in relation to these six.The chromatic opponent pairs are perceptually privative. That is, we cannot see red and green in the samepart of the visual field and the same for blue and yellow. (That a green pigment can be produced by mixingblue and yellow pigments is irrelevant.) Hering inferred that there must be a neural process which signalsred in one state and green in another (analogously for yellow and blue), hence the appelation "opponent"process. The achronmatic pair, black and white, are opposed, but not privative. We do see black and whitesimultaneously in various shades of gray. See Kaiser and Boynton (1996 23 f, 250-258) and, for a non-technical introduction to opponent theory, Wooten and Miller (1997).4 MacLaury's investigations of basic color term systems have led him to develop a theory of cognitivepoints of view, 'vantages', involving alternating attention to similarities and dissimilarities amongcognitive categories. MacLaury's (1997) interpretation of the evolution of basic color term systems isformulated largely within the vocabulary of vantage theory. Vantage theory makes broad claims in thefield of cognitive psychology (MacLaury 1997, Taylor and MacLaury 1995), which are beyond the scope ofthe present paper.5 Abbreviated below Bk, W, R, Y, G, Bu.6 Some critics of this tradition of research have misconstrued as an a priori assumption the empiricalfinding that semantic universals in color names are substantially based on the universal primary colorsensations. See, for example, Saunders and van Brakel (1988, 1995, 1997), Lucy (1996, 1997). Compare Maffi(1990a), Kay and Berlin (1997), Kay (in press). Generalization I is broader than the narrow claim of Berlinand Kay (1969) (abandoned since Kay and McDaniel 1978) that "a total universal inventory of exactlyeleven basic color categories exists from which the eleven or fewer basic color terms of any given languageare always drawn" (Berlin and Kay 1969: 2).7 We do not mean by this that basic color words are not frequently replaced by other words denoting thesame category, often borrowed words. We mean that in a given language a category once named by a basiccolor term rarely if ever becomes unnamed.8 See references in the previous note.9 Dani is the only thoroughly studied case (Heider 1972a, 1972b, Heider and Olivier 1972).10 With regard to observable living organisms, probably few languages push this tendency to the extreme ofa literally exhaustive lexical partition of the entire domain (Berlin 1992).

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 31

11 In the case of color, where the categories are gradient and overlapping, in the way treated formally byKay and McDaniel (1978), by 'partition' we intend 'fuzzy partition' as it is there defined (Kay andMcDaniel 1978: 641ff).12 For example, Kuschel and Monberg (1974), in reporting a careful ethnographic investigation of a Stage IIcolor system, make much of their impression to this effect, going so far as to entitle their report "'We don'ttalk much about color here'; a study of colour semantics on Bellona Island."13 Development due to culture contact is doubtedless the major engine of increased technological complexityin recent – perhaps in all – times. Culture contact often provides new artifacts and manufacturingtechniques, which render color a less predictable attribute of objects. Moreover, contact with a morecomplex technology is often accompanied by contact with a language whose lexicon names more distinctcolor categories (Maffi 1990b).14 See, for example, Abramov and Gordon (1994), Hård and Sivik (1981), Wooten and Miller (1997), Hardin(1988: 29f, passim). The primacy of these six color sensations has been challenged by the post-modernistsSaunders and van Brakel (1995, 1997), who reject Kay and McDaniel's (1978) "reductionist argument... [to]six basic or atomic colour categories" on the epistemological grounds, among others, that "there is noprivileged discourse in which what is true is independent of our choices, hopes and fears" (Saunders andvan Brakel 1995: 170).15 The misleading expression "fundamental neural response category" was retained in KBM.16 "Eventually someone may actually locate cells that carry out these operations" (Abramov 1997: 115).17 For example, if you wish to assess one the one hand the "distance" between a yellowish red and agreenish blue and on the other the "distance" between a yellowish green and a purplish red, there is nowell-defined, overall metric defined in color space that can tell you which of these "distances" is thegreater.18 Of the forty-seven children reported on in Dougherty (1975), eight had a term for red and lacked a termfor at least one of Y, G and Bu, while one child had terms for G and Bu but lacked a term for R (also Y).19 Not all of these differences were subjected to statistical test. A few other studies of color termacquisition were found. One reported presence and two reported absence of correlation with the full Berlinand Kay 1969 sequence, but age of acquisition for individual terms was not reported. The remainder also didnot record acquisition data for individual colors.20 In KBM two languages, Kuku-Yalanji and Murrinh-Patha, were represented as having terms for W, R,Y/G and Bk/Bu, that is, as Stage IIIY/G languages. These languages are reanalyzed in section 3, wherethey are discussed along with other languages showing strong naming for Bk, W, and R, with variablenaming elsewhere.21 The antecedents of Stage IIIY/G/Bu languages are discussed in section 3.2.22 The question marks appearing in this figure are explained in section 3.2.23 The concentration of of WCS languages on this single evolutionary path was first noted by Maffi(1988a,b).24 Since a given type may figure in more than one trajectory (e.g., type IVG/Bu appears in trajectories A, Band D), our assignment of ninety-one languages to the main line represents the maximum number of typescompatible with this trajectory, not the number of types uniquely assignable to this trajectory.25 There is independent evidence that blue is an inherently cool color (Palmer in press).26 Three languages not shown on Figure 3 are in apparent transition directly from Stage IIIBk/G/Bu toStage V.27 As may also be seen in Figure 3 (and note 26), the WCS sample does not contain any simple cases ofIIIBk/G/Bu languages, although it does contain six cases of apparent transitions either into or out of thattype.28Two of the languages in the WCS sample do not fit perfectly any of the types discussed so far, but alsoshow no evidence of the EH. Gunu (Cameroon) has terms for W, R/Y, Bk/G/Bu and Bu. It thus represents astandard Stage II system except for the presence of the blue term. The blue term is stronger than theBk/G/Bu term in the blue area, requiring that it be considered basic and therefore that Gunu be considered aviolation of the model sensu strictu. Waorani (Ecuador) is an anomalous Stage IIIG/Bu system; it contains

Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 32

terms for Bk, W/Y, R and G/Bu (rather than the standard Bk, W, R/Y and G/Bu). These two cases bring to103 (95%) the number of WCS languages which offer no support for the EH.29 And, similarly, it could have had the reduplication of a form denoting a white blossom for white or areduplication of blood for red. That is, the (hypothetical) coiner of the color term not only has to chose toform it by reduplication, but then has to chose which of several plausible bases to use. Some languageschoose blood, others fire, yet others – like Yélîdnye – choose a red bird.30 Recent unpublished data on Arrernte color terms, collected by Wilkins using the WCS stimuli, suggeststrongly that the putative Arrernte Y/G/Bu term is focused in G by all speakers and extended into both Yand Bu by a minority. One of several hypotheses consistent with the available data is that historicallythe Arrernte term now focused in green denoted a Y/G/Bu category, as reported by Spencer and Gillen(perhaps focused in green, perhaps not) and has retracted for some speakers under pressure from English.31 Maffi (1990a) raises the question whether certain Y/G/Bu (and other ) terms might not profitably beregarded as 'interstitial'.32The word 'confusion' here, and above, does not, of course, indicate that speakers are confused about how touse their language, but that the results of the WCS naming task are confused because the language does notappear to have a single, widely shared lexical strategy for naming certain regions of the color space. TheEH is, of course, about just such circumstances.33 Yélîdnye, however, does not show evidence of developing a Y/G/Bu term.34 As KBM point out, Latin had a G/Bu term viridis while Ancient Greek had a Y/G term khlôros. If thesewords were related, the situation would be comparable to that of the Salishan family. They are notrelated. The former probably comes from a PIE root denoting a surface appearance – perhaps shiny orbrilliant, the latter a PIE root related to growth (Pokorny 1948).

OFFPRINTS

The Retinex Theory of Color Vision SCIENTIFIC AMERICAN

by Edwin H. Land DECEMBER 1977 VOL 237 NO 6 P 108-128

PUBLISHED BY W. H. FREEMAN AND COMPANY 660 MARKET STREET, SAN FRANCISCO, CALIFORNIA 94104

Copyright ©1977 by Scientific American Inc. All rights reserved, Printed in the U. S. A No part of this offprint may be reproduced by any mechanical, photographic or electronic process, or in the form of a phonographic recording, nor may it be stored in a retrieval system, transmitted or otherwise copied for public or private use without written permission of the publisher.

The Retinex Theory of Color Vision

Λ retina-and-cortex system (retinex) may treat a color as a code

for a three-part report from the retina, independent of the flux

of radiant energy but correlated with the reflectance of objects

by Edwin H. Land

The scientific tradition of simplify­ing the conditions of an experi­ment has left us until recently

without a satisfactory explanation of how the eye sees color in everyday life. Paradoxically the modern technology of color photography has reinforced the belief that the colors discerned by New­ton in the spectrum are. with minor qualifications, the colors of the world around us. We know, for example, that if we use daylight color film when we take a picture in the light shed by an ordinary tungsten-filament lamp, the picture will turn out to have a strong reddish cast. That, we say. is because the rays from the tungsten filament are too "red. " never asking how we ourselves can move constantly in and out of tung­sten-lit worlds without experiencing any change in the color of familiar objects: apples, lemons, strawberries, bread, hu­man faces (the tones of which are so hard to get right on a television screen).

How, then, does the eye deal with the excess of " r e d " i n a tungsten-lit room? As I hope to demonstrate in this article. the eye. in determining color, never per­ceives the extra red because it does not depend on the flux of radiant energy reaching it. The eye has evolved to see the world in unchanging colors, regard­less of always unpredictable, shifting and uneven illumination. How the eye achieves this remarkable feat has fasci­nated me for many years.

In 1959 I described in these pages a series of experiments in which a scene created by the superposition of two black-and-white transparencies, one pro­jected through a red filler and the oth­er projected without a filter (that is, in white light), conveys to the eye nearly the gamut of colors present in the origi­nal scene [see "Experiments in Color Vi­sion. " by Edwin H. Land: SCIENTIFIC AMERICAN Offprint No. 223]. To pro­duce such "red-and-white" images the picture projected through the red filter is

taken through a red filter and the picture projected in white light is taken through a green filter. It would be expected that the superposed image on the projection screen could generate only red. white and various shades of pink. Actually one sees a picture remarkably similar to the full-color photograph reproduced on the opposite page. In the red-and-white photographic projection peppers are green, radishes and strawberries are red. the orange is orange, the lemon and bananas are pale yellow, the wood cut­ting board and knife handle are brown and the design on the plate is blue.

The challenge presented by our early red-and-white experiments led us step by step over a 20-year period to an ex­planation of how the visual system is able to extract reliable color informa­tion from the world around us. a world in which virtually every scene is lighted unevenly, in which the spectral compo­sition of the radiation falling on a scene can vary enormously and in which illu­mination as brief as a lightning flash suf­fices for the accurate identification of color. If the nature of the responses of the photoreceptors in the retina of the eye even approximated what most of us were taught in school, functioning pri­marily as intensity-level meters with peaks in three different parts of the spec­trum. we would be continually confus­ing one color with another. An object that looked yellow in one part of our field of view might look green or gray or even red when moved to a different part of the field. The fact remains that ob­jects retain their color identity under a great variety of lighting conditions. This constancy is not a minor second-order effect but is so fundamental as to call for a new description of how we sec color.

The visual pigments are photosensi­tive molecules that respond to a

wide band of light frequencies. The three pigments in the cone cells of the

retina cover the visible spectrum in three broad, overlapping curves. The pigment with a peak sensitivity at a wavelength of 440 nanometers responds in some degree to the entire lower-fre­quency half of the visible spectrum. Each of the other two pigments re­sponds to almost two-thirds of the visi­ble spectrum, the two being offset at their peaks by barely 30 nanometers. with their peak sensitivities located at 535 and 565 nanometers [see upper illus­tration on page 4].

In this discussion the names of col­ors — " r e d " , "green", "blue" and so on — will be reserved for the color sensation we have when we look at the world around us. In short, only our eyes can categorize the color of objects; spectro­photometers cannot. This point is not a trivial one because many people view­ing some of our experiments for the first time will identify something as being red or green but will then ask, as if their eyes were being fooled. "What color is it real­ly?" The answer is that the eye is not being fooled. It is functioning exactly as it must with involuntary reliability to see constant colors in a world illuminat­ed by shifting and unpredictable fluxes of radiant energy.

Since I believe the study of color in fully colored images is best begun by examining images that are completely-devoid of and completely uncomplicat­ed by the experience of color, let me describe that experience in some detail. The hypersensitive system based on the rod cells in the retina functions at light levels as much as 1. 000 times weaker than the systems based on the cone cells do, so that it is possible to answer the interesting question: What colors will one see if only the rod system is activat­ed? One procedure is to put on a pair of tightly fitting goggles equipped with neutral-density filters that reduce the in­cident light by a factor of 30. 000. After one has worn the goggles for about half

2

an h o u r objects in a r o o m i l luminated to the typical level of 20 foot-candles will b e c o m e visible. T h e effective i l lumina­t ion in the r o o m will thus be 1 / 1 , 500 foot -candle . A s o n e looks a r o u n d the r o o m the familiar co lored objects will be seen devoid of color, exhibit ing in­stead a range of l ightnesses f rom whi te to black, m u c h as they wou ld appea r

in a b lack-and-whi te p h o t o g r a p h taken th rough a green co lo r - separa t ion filter. In o the r words , the reds will appea r very da rk , the greens lighter, the blues dark . the whi tes l ight a n d the b l acks v e r y dark .

In this color less wor ld o n e f inds tha t the n a t u r e of the image is no t deter­mined by the f lux of r ad ian t energy

reach ing the eye. T h e i l luminat ion can easily be a r ranged so that there is m o r e flux f rom a region that cont inues to look very dark than there is f rom a region tha t con t inues to look very light, whe th­er these regions are three-dimensional objects or ar t i facts contr ived with a m o n t a g e of da rk and light pieces of pa­per . T h e p a r a d o x immedia te ly arises

STILL LIFE was used to make the four black-and-white images pre­sented below. The reproduction of the still life above was made by conventional processes of color photography and photoengraving to show the reader what the colors of the original objects in the scene

were. The black-and-white images were made with film-filter com­binations that closely duplicate the separate wavelength sensitivities of the four systems of photoreceptors in the retina of the eye: the three systems of cone cells and the hypersensitive system of rod cells.

BLACK-AND-WHITE IMAGES OF STILL LIFE were taken with four different film-filter combinations, creating what the author calls retinex records. The picture at the top left was taken with a film whose spectra] response was altered so that it matched the spectral sensitivity of the long-wave cone pigments in the eye. This photo­graph enables the observer to see a colorless image that approximates the image produced by the long-wave cones by themselves. The pic­

ture at the top right shows the same scene as it would be viewed by the middle-wave cone pigment The picture at the bottom left is the scene as it would be viewed by the short-wave cone pigment. The pic­ture at bottom right corresponds to the image seen by the rods. Un­like cone images, which cannot be viewed independently, images pro­duced by the rod pigment can be studied in isolation at very low light levels, without interference from much less sensitive cone systems.

3

that each of the objects, the pieces of paper for example, whether dark or light or in between, maintains its light­ness without significant change as it is moved around the room into regions of higher or lower flux. Light papers will be seen as being light and dark papers simultaneously as being dark, even with the same flux coming from each of them to the eye. Strong gradients of flux across the field will be apparent only weakly, if at all.

Furthermore, in an intricate collage of areas of various lightnesses sizes and shapes, the lightness of a given element does not change visibly as it is relocated

in any part of the collage and associated with a new arbitrary surround. When a small area is totally surrounded by a large area, the lightness of the small area will change somewhat depending on whether the large area is darker or light­er than the small one. In general, howev­er. the impressive fact is that the light­ness of a given area is not appreciably modified by the immediately surround­ing areas, nor is it modified by the still larger areas surrounding them.

Although I have been describing a col­orless world as it is seen by the hy­

persensitive receptors of rod vision, all

the observations about the stability of lightness values can readily be repro­duced with a montage of white, black and gray papers viewed at ordinary light levels. If. for example, a square of matte-surface black paper or, better still, black velvet is placed at one side of such a montage and a square of white paper is placed at the opposite side sev­eral feet away, with an assortment of light and dark papers scattered in be­tween. one can place a strong light source close enough to the black square so that it sends more radiant energy to the eye than the white square, remote from the light: yet the black square will continue to look black and the white square white. In fact, with the montage still strongly illuminated from one side either the black square or the white one can be moved to any other part of the. montage without a significant change in its appearance.

This remarkable ability of the eye to discover lightness values independent of flux, so convincingly demonstrated when only a single photoreceptor sys­tem is operating, is the rock on which a satisfactory description of color vision can be built. The first response of the visual system is for the receptors to ab­sorb the light falling on the retina. Whereas the initial signal produced in the outer segment of the receptor cell is apparently proportional to the light flux absorbed by the visual pigment, the final comprehensive response of the visual system is "lightness. " which shows little or no relation to the light flux absorbed by the visual pigment.

The processing of fluxes to generate lightnesses could occur in the retina, or in the cerebral cortex, or partially in both. Since we are uncertain of the loca­tion of the mechanisms that mediate these processes. I have coined the term retinex (a combination of retina and cor­tex) to describe the ensemble of biologi­cal mechanisms that convert flux into a pattern of lightnesses. 1 shall therefore use the term throughout this article in referring to these biological mecha­nisms. I shall also reserve the term light­ness to mean the sensation produced by a biological system. Although the rods can be stimulated at light intensities be­low the cone threshold, the cones cannot be stimulated without exciting the rods. For cones we must study the lightness images produced by each individual set of receptors using retinex photography. as I shall explain below, or learn the properties of lightness images from model calculations based on spectrora¬ diometric measurements.

Now that we know thai at low light levels an isolated receptor system

generates an image in terms of lightness that is completely free of color, might it be possible to bring one of the cone sys­tems into operation along with the hy­persensitive system, so that only the

WAVELENGTH (NANOMETERS)

NORMALIZED SPECTRAL SENSITIVITIES OF FOUR VISUAL PIGMENTS (solid lines) span the visual spectrum in overlapping curves. Curve that peaks at about 500 nanome­ters corresponds to sensitivity of rod pigment. Other three curves represent cone pigments. Bro­ken lines show sensitivities of the film-filter combinations that were selected to match the sen­sitivities of the four retinal pigments and used to make the black-and-white retinex records in the illustration at bottom of preceding page. Cone curves arc adapted from work of Paul Brown and George Wald of Harvard University. The rod curve is standard scotopic luminosity curve.

WAVELENGTH (NANOMETERS)

THRESHOLD RESPONSES OF RETINAL RECEPTORS vary by large factors. The hyper­sensitive rod system provides vision at radiance levels about 1000 times weaker than the light levels needed to activate (he cone systems. It has been shown in author's laboratory that multi­colored scenes exhibit nearly their normal range of colors when they are viewed at light levels so adjusted that only rod system and one cone system, the long-wave system, are responding.

"COLOR MONDRIAN" EXPERIMENT employs two identical dis­plays of sheets of colored paper mounted on boards four and a half feet square. The colored papers have a matte finish to minimize spec­ular reflection. Each "Mondrian" is illuminated with its own set of three projector illuminators equipped with band-pass filters and in­dependent brightness controls so that the long-wave ("red"), middle-wave ("green") and short-wave ("blue") illumination can be mixed in any desired ratio. A telescopic photometer can be pointed at any area to measure the flux, one wave band at a time, coming to the eye

from that area. The photometer reading is projected onto the scale above the two displays. In a typical experiment the illuminators can be adjusted so that the white area in the Mondrian at the left and the green area (or some other area) in the Mondrian at the right are both sending the same triplet of radiant energies to the eye. The actual ra­diant-energy fluxes cannot be re-created here because of the limita­tions of color reproduction. Under actual viewing conditions white area continues to look white and green area continues to look green even though the eye is receiving the same flux triplet from both areas.

LONG WAVE 5. 8 MIDDLE WAVE 3. 2 SHORT WAVE 1. 6 ENERGY AT EYE

(MILLIWATTS PER STERADIAN PER SQUARE METER)

LONG WAVE 5. 8 MIDDLE WAVE 3. 2 SHORT WAVE 1. 6 ENERGY AT EYE

(MILLIWATTS PER STERADIAN PER SQUARE METER)

LONG WAVE 5. 8 MIDDLE WAVE 3. 2 SHORT WAVE 1. 6 ENERGY AT EYE

(MILLIWATTS PER STERADIAN PER SQUARE METER)

IDENTICAL ENERGY FLUXES AT THE EYE provide different color sensations in the Mondrian experiments. In this example, with the illuminants from the long-wave, middle-wave and short-wave il­luminators adjusted as indicated, an area that looks red continues to look red (left), an area that looks blue continues to look blue (middle)

and an area that looks green continues to look green (right), even though all three arc sending to the eye the same triplet of long-, mid­dle- and short-wave energies. The same triplet can be made to come from any other area: if the area is white, it remains white; if the area is gray, it remains gray; if it is yellow, it remains yellow, and so on.

5

completely colorless system and one other were functioning? This two-recep­tor experiment has been carried out and provides a powerful confirmation of the ideas derived from all our binary work with red-and-white images and subse­quent ternary studies with multicolored displays seen under various illuminants. The experiment, rapidly becoming a classic, was devised by my colleagues John J. McCann and Jeanne L. Benton.

McCann and Benton illuminated a

EXPERIMENTAL ILLUMINANTS (NANOMETERS)

LEFT EYE

color display with a narrow wave band of light at 550 nanometers. The light level was raised just above the amount needed to make the display visible to the dark-adapted eye. thus ensuring that only the hypersensitive system was op­erating. They then added a second nar­row-band illuminant at 656 nanometers. with its level adjusted so that it was just sufficient to activate the long-wave re­ceptor system but not the middle-wave system. Under these conditions only two

STANDARD "WHITE" ILLUMINANT (NANOMETERS)

RIGHT EYE

COLOR-MATCHING EXPERIMENT uses a simplified Mondrian of 17 color areas (left) and a standard color reference, The Munsell Book of Color, which contains 1150 color "chips" (right). The Mondrian is illuminated with three narrow-band light sources: one at 630 nano­meters (long-wave- light), one at S30 nanometers (middle-wave light) and one at 450 nanome­ters (short-wave light). The ratio of the three illuminants can be adjusted so that the triplet of energies reflected to the eye from any chosen area will exactly equal the triplet that previously reached the eye from some other area. In this experiment five areas, gray, red, yellow, blue and green, were selected in sequence to send the same triplet of energies to the eye. In each of the five consecutive parts of this experiment the observer selected from the Munsell book the chips that came closest to matching the 17 areas of the Mondrian. The Munsell book was illuminated throughout the experiment with a constant spectral mixture of three narrow-band lights ad­justed at the outset so that the white Munsell chip appeared the "best white. " The experiment was set up so that the observers used one eye for viewing the Mondrian and the other eye for viewing chips. Gray paper with an opening was used to provide chips with a constant surround.

receptor systems, namely the rods and the long-wave cones, were receiving enough light to function.

The resulting image exhibited a re­markable range of color, enabling an observer to assign to each area in the display the same color name it would have if it were illuminated above the cone threshold. The result is reminiscent of the multicolored images produced by the red-and-white system. The demon­stration explicitly confirms our early proposition that the lightness informa­tion collected at two wave bands by sep­arate receptor systems is not averaged. point by point and area by area, but is kept distinct and is compared. We know that the rod system does not produce a colored image when the image is seen by itself, and we know that the long-wave light alone cannot produce an image with a variety of colors. The combina­tion. however, gives rise to a wide va­riety of colors, namely reds, yellows, browns, blue-greens, grays and blacks.

What, then, accounts for the color? The emergence of variegated colors can be ascribed to a process operating some­where along the visual pathway that compares the lightnesses of the separate images on two wave bands, provided by the two independent retinex systems. The two-receptor experiment makes it plausible that when three independent images constituting the lightnesses of the short-, middle- and long-wave sets of receptors are associated to give a full-colored image, it is the comparison of the respective lightnesses, region by re­gion. that determines the color of each region. The reason the color at any point in an image is essentially independent of the ratio of the three fluxes on three wave bands is that color depends only on the lightness in each wave band and lightness is independent of flux.

As we have seen, the spectral sensitivi­ties of the visual pigments overlap

broadly. If we illuminated a scene with the entire range of wavelengths to which a single visual pigment is sensitive, we would see a large variety of colors be­cause more than one retinex system would respond. With the help of filters and appropriate film emulsions, how­ever. we can isolate the lightnesses that would ordinarily be incorporated into the sensation of color. We call black-and-white photographs made for this purpose retinex records.

The photographic technique, making use of silver emulsions, performs two functions. First, the system provides spectral sensitivities that are the same as those of the visual pigments. Second, it generates black-and-white pictures for a human observer to examine. It is the hu­man visual system that converts the photographic pattern deposited in sil­ver into lightness. Ideally we should like

ILLU

MIN

AN

T

(MIL

LIW

ATT

S P

ER

S

TER

AD

IAN

PER

S

QU

AR

E M

ETE

R)

EN

ER

GY

AT

EY

E

(MIL

LIW

AT

TS

PE

R

ST

ER

AD

IAN

PE

R

SQ

UA

RE

ME

TE

R)

WAVELENGTHS (NANOMETERS)

WAVELENGTHS (NANOMETERS)

WAVELENGTHS (NANOMETERS)

WAVELENGTHS (NANOMETERS)

WAVELENGTHS (NANOMETERS)

PROPORTIONS OF NARROW-BAND ILLUMINANTS used to light the simplified Mondrian in the Munsell-chip matching experi­ments were adjusted as is shown by the bars at the lop of this illustra­tion so that five different areas of the Mondrian (indicated by arrows) sent to the observer's eye in successive matching trials the same trip­

let of energies: 5. 8 flux units of long-wave light, 3. 2 flux units of mid­dle-wave light and 1. 6 flux units of short-wave light. The illustration below shows the Munsell chips that were selected in the constant illu¬ minant to match the five Mondrian areas (gray, red, yellow, blue and green) that had sent to the eye exactly the same triplet of energies.

STA

ND

AR

D "

WH

ITE

IL

LUM

INA

NT

(M

ILLI

WA

TTS

PE

R

STE

RA

DIA

N P

ER

S

QU

AR

E M

ETER

)

MA

TCH

ING

M

UN

SE

LL

CH

IP

GRAY RED YELLOW BLUE GREEN

EN

ER

GY

AT

EY

E

(MIL

LIW

ATT

S P

ER

STE

RA

DIA

N P

ER

S

QU

AR

E M

ETE

R)

WAVELENGTHS (NANOMETERS)

WAVELENGTHS (NANOMETERS)

WAVELENGTHS (NANOMETERS)

WAVELENGTHS (NANOMETERS)

WAVELENGTHS (NANOMETERS)

MUNSELL CHIPS SELECTED BY OBSERVERS to match the five Mondrian areas that had sent identical triplets of energy to the eye are reproduced. The Munsell book was illuminated with a constant spectral mixture of narrow-band illuminants (bars at top) and the chips were viewed within a constant gray surround. The energy that

was sent to the eye by the selected Munsell chips is shown by the bars at the bottom of the illustration. It is evident that the match between the Mondrian areas and the Munsell chips is not made on the basis of the flux of radiant energy at the eye of the observer. What does cause the two areas to match is described in the illustrations that follow.

EXPERIMENTAL ILLUMINANTS STANDARD "WHITE" ILLUMINANT

LEFT EYE

RIGHT EYE

MONDRIAN MUNSELL BOOK OF COLOR

GR

AY

MONDRIAN AREA

(REFLECTANCE)

ILLUMINANT (MILLIWATTS PER STERADIAN PER SQUARE METER)

ENERGY AT EYE (MILLIWATTS PER STERADIAN PER SQUARE METER)

MUNSELL CHIP

(REFLECTANCE)

ILLUMINANT (MILLIWATTS PER STERADIAN PER SQUARE METER)

ENERGY AT EYE (MILLIWATTS PER STERADIAN PER SQUARE METER)

MATCHES

RE

D

MATCHES

YE

LLO

W

MATCHES

BLU

E

MATCHES

GR

EE

N

MATCHES

WAVELENGTHS (NANOMETERS)

WAVELENGTHS (NANOMETERS)

WAVELENGTHS (NANOMETERS)

WAVELENGTHS (NANOMETERS)

WAVELENGTHS (NANOMETERS)

WAVELENGTHS (NANOMETERS)

FURTHER ANALYSIS OF MATCHING EXPERIMENT begins to identify the basis on which the visual system makes the color match between the Mondrian area and the Munsell chip without regard to the flux each member of the pair sends to the eye. The efficiency with which a given area in the Mondrian reflects light in each of the three wave bands {first column) multiplied by the amount of energy striking that area in each of the wave bands (second column) yields

the energy triplet that reaches the eye (third column). The three col­umns at the right contain comparable data for the Munsell chips se­lected as a match for the Mondrian areas. Whereas illustration at bot­tom of the preceding page shows that the eye does not match colors using a "meter" that measures triplets of energies at the eye, this il­lustration shows that when a match is made, it is the reflectances of two areas that correspond, as is shown in first and fourth columns.

8

our observer to examine the black-and-white pattern with only one set of cones. reporting the lightnesses appropriate to that set. At any point in the black-and-white pattern, however, the reflectance is essentially the same throughout the visible spectrum. Therefore with a black-and-white photograph we stimu­late all the receptors with the same in­formation. that is. with the energies that would be absorbed by a single visual pigment. If we assume that all the reti¬ nex systems process information in an identical manner, we can propose that sending this identical information to several sets of receptors is the same as sending it to only one receptor, thereby enabling us to see what the image would look like if it were possible to isolate it.

On page 3 the reader will see three black-and-white pictures taken through retinex filters that simulate the response of the three cone pigments. The straw­berries and radishes, for example, are light on the long-wave record, darker on the middle-wave record and darkest on the short-wave record. Although the or­ange and lemon are about as dark as the strawberries and radishes on the short-wave record, they are nearly as light on the middle-wave record as they are on the long-wave record. On the printed page the distinctions are subtle. To the eye viewing an actual full-color scene the subtle distinctions provide all the in­formation needed to distinguish count­less shades and tints of every color.

After the three lightnesses of an area have been determined by the three reti­nex systems no further information is necessary to characterize the color of any object in the field of view. Any spe­cific color is a report on a trio of three specific lightnesses. For each trio of lightnesses there is a specific and unique color.

The limitations of color photography make it impossible to show the read­

er the demonstrations readily accom­plished in our laboratory, which dra­matically reveal the independence of perceived color from the flux reaching the eye. What the reader would see would be two boards four and a half feet square identically covered with about 100 pieces of paper of various colors and shapes. In order to minimize the role of specular reflectance the papers have matte surfaces and. except for black, have a minimum reflectance of at least 10 percent for any part of the visi­ble spectrum. In these displays, which we call "color Mondrians" (after the Dutch painter to whose work they bear a certain resemblance), the papers are ar­ranged so that each one is surrounded by at least five or six others of different col­ors [see top illustration on page 5 ].

Each of the identical Mondrians is il­luminated by its own set of three pro­

jectors equipped with sharply cutting band-pass filters (not retinex filters): one at 670 nanometers embracing a band of long waves, one at 540 nanometers em­bracing a band of middle waves and one at 450 nanometers embracing a band of short waves. The amount of light from each illuminating projector is controlled by a separate variable transformer. In addition the illuminating projectors have synchronized solenoid-activated shutters to control the duration of illu­mination. There is a telescopic photom­eter that can be precisely aimed at any region of either Mondrian to measure the amount of radiation reflected from any point and therefore the amount of flux reaching the eye. The output of the photometer is projected on a scale above the Mondrian. where it can be seen by those taking part in the demon­stration.

The demonstration begins with the three illuminating projectors turned on the Mondrian on the left; the Mondrian on the right remains dark. The variable transformers are set so that the entire array of papers in the left Mondrian are deeply colored and at the same time the whites are good whites. This setting is not critical. Then, using one projector at a time and hence only one wave band at a time, we measure with the telescopic photometer the energy reaching the eye from some particular area, say a white rectangle. The readings from the white area (in milliwatts per steradian per square meter) are 65 units of long-wave light. 30 units of middle-wave light and five units of short-wave light. We have now established the three energies asso­ciated with that sensation of white.

We turn off the three projectors illu­minating the color Mondrian on the left. On the right we turn on only the long-wave projector. We select a different area of unknown color and adjust the long-wave light until the long-wave en­ergy coming to the eye from the selected area is the same as the long-wave energy that a moment ago came from the white paper in the Mondrian on the left, 65 units. We turn off the long-wave projec­tor and separately adjust the transform­ers controlling the middle- and short-wave projectors, one after the other, so that the energies sent to the eye from the selected area are also the same as those that came from the white area on the left. We have not yet turned on all three light sources simultaneously, but we know that when we do so, the triplet of energies reaching the eye from the se­lected area of still unknown color will be identical with the triplet that had previ­ously produced the sensation white.

When we turn on the three illumi¬ nants, we discover that the area in the Mondrian on the right is green. We now illuminate the Mondrian on the left with its illuminants at their original settings

so that both Mondrians can be viewed simultaneously. The white area on the left continues to look white and the green area on the right continues to look green, yet both are sending to the eye the same triplet of energies: 65. 30 and five in the chosen units.

We turn off the illuminants for both Mondrians and select some other area in the left Mondrian and sequentially adjust the energies reaching the eye from it so that they are the same as the energies that originally gave rise to the sensation of white and also gave rise to the sensation of green in the right Mondrian. When we turn on all three projectors illuminating the left Mondri­an, we see that this time the selected area is yellow. The triplet of energies reach­ing our eye is the same one that had previously produced the sensations of white and green. Again, if we wish, the yellow and green can be viewed simul­taneously. with yellow on the left and green on the right.

We can continue the demonstration with other areas such as blue, gray, red and so on. It is dramatically demonstrat­ed that the sensation of color is not relat­ed to the product of reflectance times illumination, namely energy, although that product appears to be the only in­formation reaching the eye from the various areas in the Mondrians.

In order to demonstrate that the color sensations in these experiments do not involve extensive chromatic adaptation of retinal pigments the projectors are equipped with synchronized shutters so that the Mondrians can be viewed in a brief flash, a tenth of a second or less in duration. Regardless of the brevity of observation the results of the demon­strations are not altered. Thus one can say that neither chromatic adaptation nor eye motion is involved in producing the observed colors. Finally, the very essence of the design of the color Mon­drian is to obviate the significance of the shape and size of surrounding areas, of the familiarity of objects and of the memory of color. Curiously, from time to time there is a casual attempt to ad­duce what is called color constancy as an explanation of these demonstrations. Clearly color constancy is only a com­pact designation of the remarkable com­petence that is the subject of this article.

The mystery is how we can all agree with precision on the colors we see

when there is no obvious physical quan­tity at a point that will enable us to speci­fy the color of an object. Indeed, one can say the stimulus for the color of a point in an area is not the radiation from that point. The task of psychophysics is to find the nature of the stimulus for that color.

Here let us remember that what the eye does unfailingly well is to discover

9

lightness values independent of flux. We saw this to be true for a single receptor system, the rod system, operating alone and for the three cone systems operating collectively when they viewed an array of white, gray and black papers. Let us now illuminate the colored Mondrian array with light from just one of the three projectors, say the projector sup­plying long-wave light, and observe the effect of increasing and decreasing the flux by a large factor. We observe that the various areas maintain a constant rank order of lightness. If, however, we switch the illumination to a different wave band, say the middle wave band. the lightnesses of many of the areas will change: many of the 100 or so areas

will occupy a different rank order from lightest to darkest. Under the short-wave-band illuminant there will be yet a third rank order. Specifically, a red pa­per will be seen as being light in the long-wave light, darker in middle-wave light and very dark in short-wave light. A blue paper, on the other hand, will be light in short-wave light and very dark in both middle- and long-wave light. Pa­pers of other colors will exhibit different triplets of lightnesses. When we conduct­ed such experiments nearly 20 years ago. we were led inevitably to the con­clusion that the triplets of lightnesses. area by area, provided the set of con­stancies we needed to serve as the stimu­li for color, independent of flux.

It is evident that the lightnesses exhib­ited by a given piece of colored pa­per under illuminants of three different wave bands is related to the amount of energy the paper reflects to the eye at different wavelengths. Let us now exam­ine. by means of a particular experi­ment. how such reflectances can be re­lated step by step to perceived lightness­es and how. in the process, the radiant flux that reaches the eye — the ultimate source of knowledge about lightness — fi­nally becomes irrelevant to the sensa­tion of color.

In our laboratory McCann, Suzanne P. McKee and Thomas H. Taylor made

a systematic study of observers' re-

EXPERIMENTAL ILLUMINANTS

BLUE AREA IN MONDRIAN

RETINEX FILTERS RETINEX RECORDS

RA

TIO

OF

INTE

GR

ATE

D

RA

DIA

NC

ES

27. 3

100

35. 9

100

60. 7

100

34. 6

100

38. 5

100

57. 1

100

INTE

GR

ATE

D

RE

FLE

CTA

NC

ES

(P

ER

CE

NT)

27. 3

35. 9

60. 7

34. 6

38. 5

57. 1

SC

ALE

D

INTE

GR

ATE

D

RE

FLE

CTA

NC

ES

5. 8

6. 5

8. 1

6. 4

6. 7

7. 9

LONG WAVE

WHITE PAPER MIDDLE WAVE

STANDARD "WHITE" ILLUMINANT

SHORT WAVE

BLUE MUNSELL CHIP

LONG WAVE

WHITE PAPER MIDDLE WAVE

SHORT WAVE

WAVELENGTH (NANOMETERS)

ROLE OF REFLECTANCE and its psychophysical correlate, light­ness, in guiding the eye to match Munsell chips with Mondrian areas was examined with the help of retinex filter-photomultiplier combi­nations that match the spectral sensitivity of the cone pigments. Un­der each combination of illuminants (top) the integrated radiance, or flux, in each retinex wave band of a Mondrian area was compared with the integrated radiance of a sheet of white paper. The ratio of

integrated radiances yields the integrated reflectance of the Mondri­an area, expressed here in percent For the matching Munsell chip a set of ratios was similarly determined (bottom). The final step in de­riving a physical equivalent of lightness is the scaling, or spacing, of integrated reflectances to be consistent with the spacing of lightness sensations. This transformation is explained in the illustration on the opposite page. The scaled values appear in the column at the right

10

sponses to a simplified color Mondrian with areas of 17 different colors. They asked the observers to match the 17 ar­eas one at a time under different illumi¬ nants with colored squares of paper that had been selected from a standard col­or-reference book. The Munsell Book of Color and that were viewed under a constant "white" illumination.

The illuminants on the Mondrian were adjusted in five separate matching experiments so that five different areas (gray, red. yellow, blue and green) sent to the eye an identical triplet of radian­ces. The observer began by selecting a matching Munsell "chip" for each of the 17 areas in the Mondrian when the gray area in the Mondrian sent a particular triplet of energies to the eye. Another set of 17 matching Munsell chips was se­lected when the same triplet was later sent to the eye by a red area in the Mondrian. and the same was done for yellow, blue and green areas under illu­minants that supplied the same triplet of energies.

The illustrations on page 7 show the details of the experiment and the five different Munsell colors the observers selected to match the five areas when each area sent to the eye precisely the same triplet of energies. In spite of the constancy of the energy reaching one eye from the Mondrian. each observer. using the other eye. selected Munsell chips that were gray, red, yellow, blue and green.

The constant illumination used in viewing the Munsell book was a triplet of illuminants at three wavelengths that observers judged to produce the "best" white. The actual triplet of wavelengths reaching the eye from the whitest paper in the Munsell book was 11. 5 units of long-wave light. 7. 8 units of middle-wave light and 3. 3 units of short-wave light. The illuminants supplied energy in narrow bands with peaks at 630 nano­meters. 530 nanometers and 450 nano­meters. A similar triplet of narrow-band illuminants were mixed in various pro­portions to illuminate the Mondrian.

At this point the reader might ask: Would not a single gray area exhibit a pronounced change in color if the sur­rounding papers had reflected light of widely differing spectral composition? Could these changes in color account for the results of the Mondrian experi­ments? The answer to the questions is that no manipulation of surrounding pa­pers in the Mondrian is capable of mak­ing the gray paper match the red. yel­low. blue and green Munsell papers se­lected by the observers in the Mondrian experiment.

McCann, John A. Hall and I have ex­amined the matter further by repeating the Mondrian-Munsell experiment in various ways so that the average spec­tral composition of the light reaching

RE

FLE

CT

AN

CE

(P

ER

CE

NT

)

LIGHTNESS SENSATION

SENSATION OF LIGHTNESS is plotted on an equal-interval scale. Observers are shown a sheet of white paper (9) and a sheet of black paper (1) and are then asked to choose a sheet of paper whose shade of gray lies halfway between the two. The selection is the gray labeled 5. Similar selections are made to determine the locations of midpoints between 1 and 5 and be­tween 5 and 9 and so on until the equal-interval scale is filled. The end values 0 and 10 are extrapolations. The curve is then plotted by measuring the reflectances of the various papers selected by the observers. The curve makes it possible to convert values of integrated reflec­tance into values of scaled integrated reflectance, as is given in illustration on opposite page.

the eye from the Mondrian and its sur­round remains the same regardless of the spectral composition of the light needed to establish a constant triplet from area to area. We have done this in One case by surrounding the entire Mondrian with brightly colored papers selected in such a way that they exactly offset the average mixture of wave bands from the Mondrian itself and. more dramatically, by cutting the 17 ar­eas of the Mondrian apart and placing them well separated on the backgrounds of offsetting color. Neither arrangement has any significant effect on the Munsell chips chosen to match the various areas of the Mondrian.

Let us return, then, to the search for the stimulus that guides us so accurate­

ly to the correct identification of colors. If it is not a flux of radiant energy at the eye from each point in the field of view. what are the physical correlates of the lightnesses of objects on three separate

wave bands, corresponding to the spec­tral sensitivities of the cone pigments? Can such a precise physical correlate of lightness be demonstrated?

McCann, McKee and Taylor next measured the radiance, or energy at the eye. of the various Mondrian areas and of the matching Munsell chips by using a photomultiplier in conjunction with a version of the retinex filters. Since the retinex-photomultiplier combination in­tegrates the flux of radiant energy over a broad band of wavelengths, the instru­ment provides a value we call integrated radiance. McCann and his colleagues then obtained the integrated radiances from a large sheet of white paper placed under each of the experimental illumi­nants that had been used to light the Mondrian in the chip-matching experi­ments. If the integrated radiance from a Mondrian area is used as the numerator in a fraction and the integrated radiance from the white paper is used as the de­nominator. one obtains a value for in-

11

tegrated reflectance, which can be ex­pressed as a percent.

The integrated reflectances for the various Munsell chips are determined in the same manner under the constant "white" illumination. This amounts to measuring the percentage of reflectance using detectors with the same spectral sensitivity as the visual pigments. The results show that the Munsell chip cho­sen by the eye to match a given Mondri¬ an area will have approximately the same three integrated reflectances as the area. For example, the blue area in the Mondrian has a triplet of integrated re­flectances (long-, middle- and short-wave) of 27. 3. 35. 9 and 60. 7 percent. The comparable values for the matched Munsell chip are 34. 6. 38. 5 and 57. 1 percent [see illustration on page 10].

Finally, the integrated reflectances are "scaled" so that their equal spacing is consistent with the equal spacing of lightness sensations. The curve for this transformation is shown in the illustra­tion on the preceding page. Using this curve, we see that the blue area in the Mondrian has a triplet of scaled inte­

grated reflectances of 5. 8. 6. 5 and 8. 1. whereas the corresponding values for the matching Munsell chip are 6. 4. 6. 7 and 7. 9. If we study the five areas that successively sent identical triplets of en­ergies to the eye and compare their scaled integrated reflectances with those of their matching Munsell chips, we find that all the values are in excellent agree­ment. In other words, in the triplets of integrated reflectances we have identi­fied a highly accurate physical correlate of color sensations. The data fall along the 45-degree line that describes the lo­cus of perfect correlation [see illustration below].

We have sought a physical correlate for lightness, and we have found that the scaled integrated reflectances of the five areas that sent identical triplets of fluxes to our eyes are the same as those of the matching Munsell chip. This correla­tion enables us to use scaled integrated reflectances as a measured lightness equivalent. The problem now shifts to one of how the eye derives the lightness that corresponds to the reflectances of objects in each wave band.

It is one thing to measure a triplet of lightness equivalents using a retinex filter coupled to a photomultiplier: it is quite another for the eye to determine lightnesses in the unevenly lighted world without reference sheets of white paper. I described above the ability of an iso­lated receptor system — the hypersensi­tive system of rod vision — to classify ob­jects correctly according to their inher­ent reflectivity regardless of whether the objects happened to be in a brightly or a dimly lighted region of visual space. The ability of one receptor system to work in this way makes it plausible that the other three systems of normal day­time vision possess the same ability, each system viewing the world through a broad but restricted region of the spec­trum. the regions we duplicate with reti­nex filters. Each system forms a sepa­rate lightness image of the world. The images are not mixed but compared. The comparison of lightnesses at each area gives rise to the range of sensations we know as color.

How could the biological system gen­erate a hierarchy and spacing of light­ness values given only the flux from each point in a scene and knowing noth­ing about the pattern of illumination and nothing about the reflectances of objects? The scheme I am about to de­scribe is the most general we have found that surmounts these limitations: its physiological embodiment could take many forms.

Let me begin by pointing out the sig­nificance of edges in defining objects or areas in a scene. If a sheet of white paper is lighted strongly from one side, we see no discontinuity in color from one side to the other. Let us now imagine two light detectors positioned to measure the luminance from two different places on the paper. If the illumination is non­uniform. the luminances of the two places will of course be different. As the two detectors are moved closer together the luminances approach the same val­ue and the ratio of the two outputs ap­proaches unity. If, however, the two de­tectors bridge the boundary between two areas that differ abruptly in reflec­tance. such as would be the case with even a pale gray square on the white paper, the ratio of the outputs of the two detectors will approach the ratio of the two reflectances. Thus the single proce­dure of taking the ratio between two ad­jacent points can both detect an edge and eliminate the effect of nonuniform illumination. If we process the entire image in terms of the ratios of lumi­nances at closely adjacent points, we can generate dimensionless numbers that are independent of the illumination. These numbers give the ratio of reflec­tances at the edge between adjacent ar­eas: the reflectances themselves arc not yet ascertained.

In order to determine reflectances we need to relate all these ratios of reflec­

SC

ALE

D I

NT

EG

RA

TE

D R

EF

LEC

TA

NC

E O

F M

AT

CH

ING

MU

NS

ELL

CH

IPS

SCALED INTEGRATED REFLECTANCE OF AREAS IN MONDRIAN

AGREEMENT IN SCALED INTEGRATED REFLECTANCES between Mondrian areas and Munsell chips chosen to match them is summarized for all three wave-band systems. The scaled integrated reflectances of five Mondrian areas and matching Munsell chips were deter­mined as is described in illustration on page 10. In this graph triplets of scaled integrated re­flectances of five Mondrian areas that sent identical fluxes to the eye are plotted against scaled integrated reflectances of Munsell chips chosen to match them. Although the dots collectively represent correspondence for all three retinex wave bands, any particular dot denotes the de­gree of correspondence on one retinex wave band between a Mondrian area and a Munsell chip. Close correspondences show that scaled integrated reflectance is physical correlate of the sensation "lightness, " showing precision with which a triplet of lightnesses determines color.

12

EV

EN

ILL

UM

INA

TIO

N (

RE

LAT

IVE

UN

ITS

)

LIGHT FLUX

RATIO AFTER

THRESHOLD TEST

SEQUENTIAL PRODUCT

THE EYE'S METHOD OF DISCOVERING LIGHTNESS in com­plex images remains to be established. An efficient and physiological­ly plausible scheme is depicted in this illustration and the one below. The numbers inside the schematic Mondrian represent the long-wave integrated radiances coming from each area of a display that is even­ly lighted. The long-wave retinex system independently "measures" the long-wave integrated radiance, point by point, as if it were doing so along an arbitrary pathway {color). The flux at each successive closely spaced pair of points is converted into a ratio. This ratio is sub­jected to a threshold test: any ratio to he regarded as a change must vary from unity by more than some email threshold amount (plus or minus. 003 in the computer program). If the ratio docs not vary from unity by this amount, it is regarded as being "unchanged" and is set to equal unity. A second threshold-tested ratio along the same pathway is multiplied by the first ratio to give a sequential product that is both

the model's response for that point and the signal sent along to be multiplied by the next ratio. When the path crosses an edge between two lightnesses, there is a sharp change in the threshold-tested ratio and hence a similar change in the sequential product Here the path is started in the white area, where the flux of radiant energy is 100. By the time the path reaches the brown area at the lower right the product is . 18. The retinex system has thus determined that the brown area reflects 18 percent as much long-wave energy as the white area. Any other path ending in the brown area would yield the same result as long as it had been through the white area. By averaging the re­sponses for each area, as computed by many arbitrary paths, the long­wave retinex system arrives at a single reflectance value for each area, which designates perceived lightness. Middle- and short-wave retinex systems compute their own sets of lightness values. Compari­son of triplet of lightnesses for each area provides sensation of color.

GR

AD

IEN

T I

LLU

MIN

ATI

ON

(R

ELA

TIV

E U

NIT

S)

LIGHT FLUX

RATIO AFTER

THRESHOLD TEST

SEQUENTIAL PRODUCT

MORE REALISTIC CASE OF GRADED ILLUMINATION is han­dled equally well by the sequential-product method to arrive at the same reflectance value of . 18 for the brown area at the end of the path, even though here the long-wave retinex system receives as much flux from the middle of the brown area {57) as it does from the middle

of the white area {57). The scheme hence provides a means for arriv­ing at computed reflectance independent of flux and without resort to white cards as standards. Precise values of light flux along pathway in this diagram were derived from a computer program that works with 75 values between every two values printed within Mondrian.

13

t ances in the field of view. G iven the ra t io of luminances a t the edge be tween a first a rea and a second one , we mul t i ­ply i t by the r a t io of l uminances at the edge be tween the second a r ea and a third. Th i s p roduc t of r a t ios a p p r o a c h e s the r a t io of reflectances be tween the first and third areas , r egard less of the dis­t r ibut ion of i l luminat ion . Similar ly , we can ob ta in the r a t io of reflectances of any two a reas in an i m a g e , howeve r re­m o t e they are from e a c h o ther , by mul t i ­plying the ra t ios of all t he boundar ies between the s tar l ing a r ea and the r e m o t e a rea . We can a l so establ ish t he r a t io of the reflectance of any a r ea on the p a t h by tapping off the sequent ia l p roduc t reached at that a r ea [see illustrations on preceding page].

We are now coming close to t he an­swer to the ques t ion : H o w can the

eye ascer ta in the reflectance of an a r ea wi thout in effect p lac ing a c o m p a r i s o n s t andard next to t he a r e a ? T h e sequen­tial p r o d u c t can be used as a subst i tu te for the p l acemen t of t w o a r ea s adjacent to each other , thus defining a p h o t o m e t ­r ic opera t ion feasible for t he eye .

T h e r emain ing task is to suggest h o w the eye can discover the a r e a of high­est reflectance in the field of view and then decide whe the r that a rea is ac tua l ­ly white or some o the r co lo r . In the m o d ­e l we have p roposed , sequent ia l p rod­ucts are c o m p u t e d a long m a n y a rb i t r a ry pa thways that w a n d e r t h r o u g h the two-dimensional a r r ay of energies on the mode l ' s "ret ina. " Since t he p a t h w a y s

COLOR "SOLID" shows the location of all perceivable colors, including white and black, in a three-dimensional color space construct­ed according to the author's retinex theory. The position of a color in this space is deter­mined not by the triplet of energies at a point but by the triplet of lightnesses computed by the eye for each area. The color photograph at the top left shows the location of represent­ative colors throughout the space. The direc­tion of increasing lightness along each axis is shown by the arrows. The three black-and-white photographs of the color solid were tak­en with retinex filter-film combinations. They show the lightness values of the representative colors as they would be perceived separately by the eye's long-wave (top), middle-wave (middle) and short-wave (bottom) visual pig­ments. The set of 10 color pictures at the right represents horizontal planes cut through the three-dimensional color space. Each plane is the locus of colors possible with a constant short-wave lightness. For example, the fifth plane from the bottom shows the variety of color sensations from all possible long- and middle-wave lightness values when those val­ues are combined with a short-wave lightness of 5. The colored squares are samples taken from The Munsell Book of Color. In general the blank areas on each plane represent re­gions where colors could be produced only by fluorescent dyes, if they were produced at all.

can begin anywhere, not just in regions of the highest reflectance, the first value in any sequence is arbitrarily assumed to be 100 percent. Because of this delib­erately adopted fiction the sequential product becomes greater than unity whenever the path reaches an area whose reflectance is higher than that of the starting area.

The attainment of a sequential prod­uct greater than unity indicates that the sequence should be started afresh with the new area of high reflectance taken as being 100 percent. This procedure is the heart of the technique for finding the highest reflectance in the path. After the path reaches the highest reflectance in the scene, each of the sequential prod­ucts computed thereafter becomes a fraction of the highest value. A satisfac­tory computer program has been de­signed to study the number of paths. their lengths and convolutions, the threshold values for recognizing edges and. perhaps most important, how to utilize all the pathways starting in all areas.

The biological counterpart of this program is performed in undetermined parts of the pathway between the retina and the cortex. The process that corre­sponds to computing sequential prod­ucts does not involve the averaging of areas or the averaging of flux. It does. however, call for an arithmetic that ex­tends over the entire visual field. Fur­thermore. since the relevant phenomena are seen in a brief pulse of light, all the computations and conclusions about lightness must be carried out in a frac­tion of a second without dependence on eye movement. With a single pulse, eye movement, by definition, is not neces­sary. With continuous illumination the normal quick motions of the eye proba­bly serve to maintain the freshness of the process.

With our computer model we can ob­tain a triplet of lightnesses for each area in the color Mondrian that corresponds closely to the lightnesses one would measure with a combined retinex filter and photomultiplier. The color corre­sponding to any given triplet can be vi­sualized with the aid of the color "sol­id" we have built, in which the Munsell colors are located in three dimensions in "lightness-color space" according to their lightness values measured in three wave bands through retinex filters [see illustration on page 14].

In normal images the sensation of white light will be generated by any area that is placed at the top of the lightness scale by all three retinex systems. On the other hand, an area that stands at the top of only two of the three lightness scales will be seen as some other color. Hence an area that is at the top of the lightness scale in the long- and middle-wave sys­tems but is surpassed in lightness by some other area in the short-wave sys­

tem will be seen not as white but as yel­low. A similar intercomparison of trip­lets of lightnesses at the same place within each scene provides the sensation of color, area by area, in spite of unpre­dictable variations in illumination.

If one looks at black-and-white photo­graphs taken through retinex filters, one sees a dramatic difference in lightness for most objects between the photo­graph representing the short-wave sys­tem and either of the photographs repre­senting the other two systems. And yet it is the comparatively small differences between the long-wave and the middle-wave lightnesses that are responsible for the experience of vivid reds and greens.

Such reliable and sensitive respon­siveness to small lightness differences provides the basis for the colors seen under anomalous conditions far re­moved from those the eye has evolved to see. Two examples of interest are the color of a spot of light in a total sur­rounding area devoid of light and the spectrum of colors produced by a prism.

One can readily measure the flux at the eye from a spot of light in a void. By

FLU

X (

RE

LAT

IVE

UN

ITS

)

changing the flux it is possible to esti­mate the corresponding change in per­ceived lightness. What one finds is that the estimated lightness changes only slowly with enormous changes in flux. For example, decreasing the flux by a very large amount will be seen as a very small reduction in lightness. If the spot of light is composed of a narrow band of long wavelength, say 600 nanometers. one can expect all three cone receptors to absorb the radiation in some degree. but significantly more radiation will be absorbed by the long-wave cones than by the other two kinds. When the three values are read on a scale of perceived lightness, the three lightnesses are 9 on the long-wave system. 8. 5 on the mid­dle-wave system and 7. 5 on the short­wave system [see illustration on this page]. This combination of lightnesses is seen as a light reddish orange, a color not commonly perceived under ordinary conditions unless the surfaces are fluo­rescent. The spectrum, a strikingly anomalous display, can be regarded as a series of three laterally displaced contin­uous gradients involving both the prop-

LIGHTNESS SENSATION

SPOT OF LIGHT IN A VOID, that is, a single spot of narrow-band light viewed in an other­wise totally dark environment, has a color that would seem to depend solely on Ms wavelength. The color can also be explained, however, by the retinex theory in terms of lightness as per­ceived by the eye's three receptor systems. Psychophysical measurements show that when the eye is presented with a spot of light in a void, the perceived lightness is changed only slightly by very large changes in flux, as is indicated by the straight line. For example, if the spot is com­posed of a narrow-wavelength band centered, say, at 600 nanometers, the three cone pigments will absorb the flux in quite different amounts because of the shape of their absorption curves. In arbitrary units the long-wave pigment might absorb 80 units, the middle-wave pigment 20 units and the short-wave pigment a few tenths of a unit at most. If these ratios are plotted on the spot-in-a-void curve, the corresponding lightness values are 9 for the long waves, 8. 5 for the middle waves and 7. 5 for the short This combination of lightnesses is perceived as a light reddish orange, not ordinarily seen under normal conditions unless surfaces are fluorescent

15

erties of spots and the properties of ar­eas. From these properties it is possible to predict the colors of the spectrum. whereas it is not possible, as we have seen, to attribute a specific spectral com­position to the radiance from a colored area in everyday life.

Perhaps the first observation pointedly relevant to the mechanism of color

formation in images is not Newton's spectrum but the. phenomenon of col­ored shadows, described in 1672 by Otto von Guericke. "This is how it happens. " he wrote, "that in the early morning twi­light a clear blue shadow can be pro­duced upon a white piece of paper [by holding] a linger or other object. . . be­tween a lighted candle and the paper be­neath. " This important experiment, we now know, depicts an elementary exam­ple of generating three different light­nesses on the three receptor systems. A diagram of this experiment with long­wave ("red") light and white light ap­pears below. Here the color of the shad­ow is blue-green. The diagram shows that the triplet of lightnesses in the shad­ow corresponds to the blue-green color one would predict for it from its posi­

tion in lightness-color space. One can now understand the red-and-

white images of our early work as a pro­cedure that carries the colored shadow to a richly variegated family of colors no longer in shadows but in images. The colors seen in a red-and-white projec­tion can be readily predicted by extend­ing the analysis followed in predicting the color of von Guericke's shadow. To demonstrate this point we reproduce on page 17 the "red" and "green" separa­tion images used in making a red-and-white multicolored projection. (In this demonstration no attempt is made to re­produce the colors seen in the actual multicolored image. ) The red-and-white projection was photographed through long-, middle- and short-wave retinex film-filter combinations. The three im­ages are reproduced below the pair of long- and middle-wave separation im­ages that were superposed to make the red-and-white image. The significant point is that when the eye views the red-and-white images on the screen with its own retinex system, it is provided with a triplet of lightnesses for each part of the scene that resembles the triplet it would obtain if it viewed the original scene di­

rectly. In this important meeting point of the blue-green shadows with the col­ored images, provided by the red-and-white display, the extended taking and multiplication of ratios determine the lightness of each small area. Finally, all these principles are applied in everyday ternary vision, which creates a distinct lightness image for each of the three sen­sitive systems and compares them in or­der to generate color.

The train of interlocking concepts and experiments started 25 years ago

with the observation that the relative en­ergies of the red-and-white projectors can be altered without changing the names of the various colors. This obser­vation negated the simplistic explana­tion in terms of contrast, fatigue and surround and led to the fundamental concept of independent long- and short­wave image forming systems that ulti­malely evolved to the concept of three independent retinex systems and to the Mondrian demonstration. The concept of the percentage of available light on each wave band as a determining vari­able and the technique of measuring it evolved to the concept that lightnesses

ADSORBED FLUX (ARBITRARY UNITS)

BLUE-GREEN SHADOW FROM "WHITE"

ILLUMINATOR

FROM LONG-WAVE ("RED") ILLUMINATOR

(SCATTER)

FROM TWO ILLUMINATORS

COMBINED LIGHTNESS COLOR

SH

OR

T-W

AV

E

RE

CE

PTO

R

SY

ST

EM

100

MID

DLE

-WA

VE

R

EC

EP

TOR

S

YS

TE

M

100

LON

G-W

AV

E

RE

CE

PTO

R

SY

ST

EM

100

"WHITE" ILLUMINATOR

LONGWAVE ("RED")

ILLUMINATOR

BLUE-GREEN COLORED SHADOW is seen when a hand or some other object is placed in the beam of a projector that is sending long-wovc ("red") light to a screen while the screen is illuminated by a beam of white light. The author regards Otto von Guericke's description in 1672 of seeing colored shadows made by candlelight as the first observation pointedly relevant to the mechanism of image and color formation. In the analysis at the right it is assumed that one projector sends white light to the screen. The other projector, equipped with a red filter, sends only long wavelengths to the screen. Assume that the white light contributes 100 arbitrary units of flux to each of the short-, middle- and long-wave receptors. The long-wave flux is absorbed by the three receptor systems in different proportions: 100 units are

absorbed by the long-wave system, SO by the middle-wave system and five by the short-wave system. (A small amount of scattered long­wave light also appears in the shadow. ) The third column of boxes shows the combined amounts of flux from both sources absorbed by each receptor system. The fractions represent the ratio at edges of the flux from within the shadow divided by the flux from outside. The fourth column shows the lightness on each receptor system. The lightness of the lightest place in the scene for each receptor system will be near the top of the lightness scale, being determined by the flux of radiant energy in the same way that a spot has its lightness determined by flux. Triplet of lightnesses within the shadow falls in the region of color space that the eye perceives as being blue-green.

1 6

maintain an independent rank order on long- and short-wave bands. This mea­suring technique in turn evolved from a projected black-and-white image to an arrangement of colored papers in the color Mondrian. The manifest stability and constancy of the lightnesses of all the papers of the Mondrian when a sin­gle wave band illuminates it with vary­ing intensity dramatizes the concept that every colored paper has three reflec­tances on three wave bands and that these reflectances are somehow connect­ed with the biological characteristic: lightnesses.

Ablack-and-white Mondrian taught that nonuniformity of illumination.

size and shape of area and length of edges were basically irrelevant to light­

ness. What was needed was a far-reach­ing, edge-reading arithmetic: the se­quential product of ratios at edges. For the color Mondrian the ratio at edges was early recognized as requiring a ratio of the integrals of the product at each wavelength of the absorbance of the cone pigment times the reflectance of the colored paper times the illuminants. Separate integrals were taken over the wave bands of the three cone pigments. In a long series of binocular com­parison-and-selection observations the quantity satisfying the integral was shown to be impressively well correlat­ed with lightness, particularly after the realization that the scale, or spacing, of the reflectance integral should be made to correspond with the spacing of the biological quantity lightness. This led to the designation "scaled integrated re­

flectance" as the external partner to which the retinex system relates the in­ternal partner: constructed lightness.

Color can be arranged in the lightness solid with long-, middle- and short-wave axes of lightness. All visible colors re­side in this solid independent of flux. each color having a unique position giv­en by the three axial values of lightness. It should be remembered that the reality of color lies in this solid. When the col­or Mondrian is nonuniformly illuminat­ed. photographed and measured, reflec­tance in the photograph no longer corre­lates with the color but the lightness docs. The three sets of ratios of integrals at edges and the product of these inte­grals within a set emerge as the physical determinants in the partnership between the biological system and areas in the external world.

LONG-WAVE ("RED") SEPARATION RECORD MIDDLE-WAVE ("GREEN") SEPARATION RECORD

LONG-WAVE RETINEX RECORD MIDDLE-WAVE RETINEX RECORD SHORT-WAVE RETINEX RECORD

RETINEX RECORDS OF RED-AND-WHITE projections show that red-and-white images produce a triplet of lightnesses for each part of the scene that are consistent with the observed color sensa­tions. The two photographs in the top half of this illustration are re­productions of the long-wave {left) and middle-wave (right) separa­tion records taken of the original still life. The long-wave record was projected onto a screen with a long-wave (red) filter in the beam of light. The middle-wave record was projected in superposition onto the same screen in the light of a tungsten-filament lamp. Three reti­nex photographs were then taken of projected images appearing on

screen. The retinex records are reproduced in the bottom part of the illustration: long-wave at the left, middle-wave in the middle and short-wave at the right. The colors seen in red-and-white projections are those one would expect from their triplets of lightnesses. The ap­ple is light on the long record and darker in the middle and short rec­ords. The orange is lightest on the long record, intermediate on the middle record and darkest on the short. It is impressive that with his own retinex systems the observer can see a blue cup, a brown straw basket and pale yellow bananas with lightness differences so small as to challenge photoengraving process used to reproduce photographs.

17

The Author

EDWIN H. LAND is chairman of the board, director of research and chief ex­ecutive officer of the Polaroid Corpora­tion. Born in 1909. he attended Harvard College, where he developed a new type of polarizing filter in the form of an ex­tensive synthetic sheet. In 1937 he founded Polaroid for research in the new field of applied polarization, and in 1944 he began his pioneering work in the development of "instant" photogra­phy. His one-step photographic process was first demonstrated to the Optical So­ciety of America in February. 1947, and was made available to the public at the end of 1948. Land has received 14 hon­orary degrees, has held visiting academ­ic appointments at Harvard and is cur­rently Institute Professor (Visiting) at the Massachusetts Institute of Technol­ogy. From 1960 to 1973 he was consul¬ tant-at-large to the President's Science Advisory Committee, and in 1967 he re­ceived the National Medal of Science. This year, on the occasion of his 500th U. S. patent, he was elected to the Na­tional Inventors Hall of Fame. Land has pursued his lively interest in the mecha­nisms of color vision for the past 25 years.

Bibliography COLOR VISION AND THE NATURAL IM­

AGE: PART I. Edwin H. Land in Pro­ceedings of the National Academy of Sciences. Vol. 45, No. 1. pages 115-129: January, 1959.

COLOR VISION AND THE NATURAL IM­AGE: PART II. Edwin H. Land in Pro­ceedings of the National Academy of Sciences, Vol. 45, No. 4. pages 636-644: April. 1959.

INTERACTION OF THE LONG-WAVE CONES AND THE RODS TO PRODUCE COLOR SENSATIONS. John J. McCann and Jeanne L. Benton in Journal of the Optical Society of America, Vol. 59, No. 1. pages 103-107: January. 1969.

LIGHTNESS AND RETINEX THEORY. Ed­win H. Land and John J. McCann in Journal of the Optical Society of Ameri­ca, Vol. 61 . No. 1. pages 1-11; Janu­ary, 1971.

The Cover The pattern on the cover was used in experiments testing Edwin H. Land's re­tinex theory of color vision. Because the pattern bears a resemblance to the works of the Dutch painter Piet Mondri¬ an. Land refers to this display and simi­lar ones as Mondrians. In more elabo­rate examples (see top illustration on page 5) perhaps 100 pieces of paper of vari­ous colors and sizes are mounted on large boards and so arranged that each piece of paper is surrounded by at least five or six other pieces of different col­ors. In a typical demonstration the Mondrian is illuminated by projectors that provide adjustable amounts of radi­ant energy in three wave bands: long ("red"), middle ("green") and short ("blue"). With the proper selection of the mixture of illuminants falling on the Mondrian the radiant flux reaching the eye from any selected area can be made to match the flux that had previously reached the eye from a totally different area. In the first instance the selected area could have been red: in the second instance it could have been green. With the same flux of energy reaching the eye the two areas will still be seen as red and green. (Cover photograph by Julius J. Scarpetti)

Frequently Asked Questions about Color

Charles Poynton

[email protected]

This FAQ is intended to clarify aspects of color that are important to color image coding, computer graphics, image processing, video, and the transfer of digital images to print.

I assume that you are familiar with intensity, luminance (CIE Y), light-ness (CIE L*), and the nonlinear relationship between CRT voltage and intensity (gamma). To learn more about these topics, please read the companion Frequently Asked Questions about Gamma before starting this.

This document is available on the Internet from Toronto at<http://www.poynton.com/PDFs/ColorFAQ.pdf>

I retain copyright to this note. You have permission to use it, but you may not publish it.

Table of Contents

1 What is color? 3

2 What is intensity? 3

3 What is luminance? 3

4 What is lightness? 4

5 What is hue? 4

6 What is saturation? 4

7 How is color specified? 4

8 Should I use a color specification system for image data? 5

9 What weighting of red, green and blue corresponds to brightness? 5

10 Can blue be assigned fewer bits than red or green? 6

11 What is “luma”? 6

12 What are CIE XYZ components? 7

13 Does my scanner use the CIE spectral curves? 7

14 What are CIE x and y chromaticity coordinates? 7

15 What is white? 8

16 What is color temperature? 8

17 How can I characterize red, green and blue? 9

1997-03-02a Charles A. Poynton. All rights reserved. 1 of 24

2 Frequently Asked Questions About Colour

18 How do I transform between CIE XYZ and a particular set of RGB primaries? 9

19 Is RGB always device-dependent? 10

20 How do I transform data from one set of RGB primaries to another? 10

21 Should I use RGB or XYZ for image synthesis? 11

22 What is subtractive color? 11

23 Why did my grade three teacher tell me that the primaries are red, yellow and blue? 11

24 Is CMY just one-minus-RGB? 12

25 Why does offset printing use black ink in addition to CMY? 12

26 What are color differences? 13

27 How do I obtain color difference components from tristimulus values? 14

28 How do I encode Y'PBPR components? 14

29 How do I encode Y'CBCR components from R'G'B' in [0, +1]? 15

30 How do I encode Y'CBCR components from computer R'G'B' ? 15

31 How do I encode Y'CBCR components from studio video? 16

32 How do I decode R'G'B' from PhotoYCC ? 17

33 Will you tell me how to decode Y'UV and Y'IQ? 17

34 How should I test my encoders and decoders? 17

35 What is perceptual uniformity? 18

36 What are HSB and HLS? 19

37 What is true color? 19

38 What is indexed color? 20

39 I want to visualize a scalar function of two variables. Should I use RGB values corresponding to the colors of the rainbow? 21

40 What is dithering? 21

41 How does halftoning relate to color? 21

42 What’s a color management system? 22

43 How does a CMS know about particular devices? 22

44 Is a color management system useful for color specification? 22

45 I’m not a color expert. What parameters should I use to code my images? 23

46 References 23

47 Contributors 24

2

Frequently Asked Questions About Color 3

1 What is color? Color is the perceptual result of light in the visible region of the spectrum, having wavelengths in the region of 400 nm to 700 nm, incident upon the retina. Physical power (or radiance) is expressed in a spectral power distri-bution (SPD), often in 31 components each representing a 10 nm band.

The human retina has three types of color photoreceptor cone cells, which respond to incident radiation with somewhat different spectral response curves. A fourth type of photoreceptor cell, the rod, is also present in the retina. Rods are effective only at extremely low light levels (colloquially, night vision), and although important for vision play no role in image reproduction.

Because there are exactly three types of color photoreceptor, three numer-ical components are necessary and sufficient to describe a color, providing that appropriate spectral weighting functions are used. This is the concern of the science of colorimetry. In 1931, the Commission Interna-tionale de L’Éclairage (CIE) adopted standard curves for a hypothetical Standard Observer. These curves specify how an SPD can be transformed into a set of three numbers that specifies a color.

The CIE system is immediately and almost universally applicable to self-luminous sources and displays. However the colors produced by reflec-tive systems such as photography, printing or paint are a function not only of the colorants but also of the SPD of the ambient illumination. If your application has a strong dependence upon the spectrum of the illu-minant, you may have to resort to spectral matching.

Sir Isaac Newton said, “Indeed rays, properly expressed, are not colored.” SPDs exist in the physical world, but colour exists only in the eye and the brain.

Berlin and Kay [1] state that although different languages encode in their vocabularies different numbers of basic color categories, a total universal inventory of exactly eleven basic color categories exists from which the eleven or fewer basic color terms of any given language are always drawn. The eleven basic color categories are WHITE, BLACK, RED, GREEN, YELLOW, BLUE, BROWN, PURPLE, PINK, ORANGE, and GRAY.

2 What is intensity? Intensity is a measure over some interval of the electromagnetic spectrum of the flow of power that is radiated from, or incident on, a surface. Inten-sity is what I call a linear-light measure, expressed in units such as watts per square meter.

The voltages presented to a CRT monitor control the intensities of the color components, but in a nonlinear manner. CRT voltages are not proportional to intensity.

3 What is luminance? Brightness is defined by the CIE as the attribute of a visual sensation according to which an area appears to emit more or less light. Because bright-ness perception is very complex, the CIE defined a more tractable quan-tity luminance which is radiant power weighted by a spectral sensitivity function that is characteristic of vision. The luminous efficiency of the Stan-dard Observer is defined numerically, is everywhere positive, and peaks

4 Frequently Asked Questions About Color

at about 555 nm. When an SPD is integrated using this curve as a weighting function, the result is CIE luminance, denoted Y.

The magnitude of luminance is proportional to physical power. In that sense it is like intensity. But the spectral composition of luminance is related to the brightness sensitivity of human vision.

Strictly speaking, luminance should be expressed in a unit such as candelas per meter squared, but in practice it is often normalized to 1 or 100 units with respect to the luminance of a specified or implied white reference. For example, a studio broadcast monitor has a white reference whose luminance is about 80 cd•m-2, and Y = 1 refers to this value.

4 What is lightness? Human vision has a nonlinear perceptual response to brightness: a source having a luminance only 18% of a reference luminance appears about half as bright. The perceptual response to luminance is called Lightness. It is denoted L* and is defined by the CIE as a modified cube root of lumi-nance:

Yn is the luminance of the white reference. If you normalize luminance to reference white then you need not compute the fraction. The CIE defini-tion applies a linear segment with a slope of 903.3 near black, for (Y/Yn) ≤ 0.008856. The linear segment is unimportant for practical purposes but if you don’t use it, make sure that you limit L* at zero. L* has a range of 0 to 100, and a “delta L-star” of unity is taken to be roughly the threshold of visibility.

Stated differently, lightness perception is roughly logarithmic. An observer can detect an intensity difference between two patches when their intensities differ by more than one about percent.

Video systems approximate the lightness response of vision using R’G’B’ signals that are each subject to a 0.45 power function. This is comparable to the 1⁄ 3 power function defined by L* .

5 What is hue? According to the CIE [2], hue is the attribute of a visual sensation according to which an area appears to be similar to one of the perceived colours, red, yellow, green and bue, or a combination of two of them. Roughly speaking, if the dominant wavelength of an SPD shifts, the hue of the associated color will shift.

6 What is saturation? Again from the CIE, saturation is the colourfulness of an area judged in propor-tion to its brightness. Saturation runs from neutral gray through pastel to saturated colors. Roughly speaking, the more an SPD is concentrated at one wavelength, the more saturated will be the associated color. You can desaturate a color by adding light that contains power at all wavelengths.

7 How is color specified? The CIE system defines how to map an SPD to a triple of numerical components that are the mathematical coordinates of color space. Their function is analagous to coordinates on a map. Cartographers have different map projections for different functions: some map projections preserve areas, others show latitudes and longitudes as straight lines. No

LY

Y

Y

Yn n

* .=

− <116 16 0 008856

13

;

24

Frequently Asked Questions About Color 5

single map projection fills all the needs of map users. Similarly, no single Color system fills all of the needs of color users.

The systems useful today for color specification include CIE XYZ, CIE xyY, CIE L*u*v* and CIE L*a*b*. Numerical values of hue and satura-tion are not very useful for color specification, for reasons to be discussed in section 36.

A color specification system needs to be able to represent any color with high precision. Since few colors are handled at a time, a specification system can be computationally complex. Any system for color specifica-tion must be intimately related to the CIE specifications.

You can specify a single “spot” color using a color order system such as Munsell. Systems like Munsell come with swatch books to enable visual color matches, and have documented methods of transforming between coordinates in the system and CIE values. Systems like Munsell are not useful for image data. You can specify an ink color by specifying the proportions of standard (or secret) inks that can be mixed to make the color. That’s how PANTONE works. Although widespread, it’s propri-etary. No translation to CIE is publicly available.

8 Should I use a color specification system for image data?

A digitized color image is represented as an array of pixels, where each pixel contains numerical components that define a color. Three compo-nents are necessary and sufficient for this purpose, although in printing it is convenient to use a fourth (black) component.

In theory, the three numerical values for image coding could be provided by a color specification system. But a practical image coding system needs to be computationally efficient, cannot afford unlimited precision, need not be intimately related to the CIE system and generally needs to cover only a reasonably wide range of colors and not all of the colors. So image coding uses different systems than color specification.

The systems useful for image coding are linear RGB, nonlinear R’G’B’ , nonlinear CMY, nonlinear CMYK, and derivatives of nonlinear R’G’B’ such as Y’CBCR. Numerical values of hue and saturation are not useful in color image coding.

If you manufacture cars, you have to match the color of paint on the door with the color of paint on the fender. A color specification system will be necessary. But to convey a picture of the car, you need image coding. You can afford to do quite a bit of computation in the first case because you have only two colored elements, the door and the fender. In the second case, the color coding must be quite efficient because you may have a million colored elements or more.

For a highly readable short introduction to color image coding, see DeMarsh and Giorgianni [3]. For a terse, complete technical treatment, read Schreiber [4].

9 What weighting of red, green and blue corresponds to brightness?

Direct acquisition of luminance requires use of a very specific spectral weighting. However, luminance can also be computed as a weighted sum of red, green and blue components.

6 Frequently Asked Questions About Color

If three sources appear red, green and blue, and have the same radiance in the visible spectrum, then the green will appear the brightest of the three because the luminous efficiency function peaks in the green region of the spectrum. The red will appear less bright, and the blue will be the darkest of the three. As a consequence of the luminous efficiency func-tion, all saturated blue colors are quite dark and all saturated yellows are quite light. If luminance is computed from red, green and blue, the coeffi-cients will be a function of the particular red, green and blue spectral weighting functions employed, but the green coefficient will be quite large, the red will have an intermediate value, and the blue coefficient will be the smallest of the three.

Contemporary CRT phosphors are standardized in Rec. 709 [9], to be described in section 17. The weights to compute true CIE luminance from linear red, green and blue (indicated without prime symbols), for the Rec. 709, are these:

This computation assumes that the luminance spectral weighting can be formed as a linear combination of the scanner curves, and assumes that the component signals represent linear-light. Either or both of these conditions can be relaxed to some extent depending on the application.

Some computer systems have computed brightness using (R+G+B)/3. This is at odds with the properties of human vision, as will be discussed under What are HSB and HLS? in section 36.

The coefficients 0.299, 0.587 and 0.114 properly computed luminance for monitors having phosphors that were contemporary at the introduction of NTSC television in 1953. They are still appropriate for computing video luma to be discussed below in section 11. However, these coeffi-cients do not accurately compute luminance for contemporary monitors.

10 Can blue be assigned fewer bits than red or green?

Blue has a small contribution to the brightness sensation. However, human vision has extraordinarily good color discrimination capability in blue colors. So if you give blue fewer bits than red or green, you will introduce noticeable contouring in blue areas of your pictures.

11 What is “luma”? It is useful in a video system to convey a component representative of luminance and two other components representative of color. It is impor-tant to convey the component representative of luminance in such a way that noise (or quantization) introduced in transmission, processing and storage has a perceptually similar effect across the entire tone scale from black to white. The ideal way to accomplish these goals would be to form a luminance signal by matrixing RGB, then subjecting luminance to a nonlinear transfer function similar to the L* function.

There are practical reasons in video to perform these operations in the opposite order. First a nonlinear transfer function – gamma correction – is applied to each of the linear R, G and B. Then a weighted sum of the nonlinear components is computed to form a signal representative of luminance. The resulting component is related to brightness but is not CIE luminance. Many video engineers call it luma and give it the symbol Y’. It is often carelessly called luminance and given the symbol Y. You

Y R G B709 0 2125 0 7154 0 0721= + +. . .

24

Frequently Asked Questions About Color 7

must be careful to determine whether a particular author assigns a linear or nonlinear interpretation to the term luminance and the symbol Y.

The coefficients that correspond to the “NTSC” red, green and blue CRT phosphors of 1953 are standardized in ITU-R Recommendation BT. 601-4 (formerly CCIR Rec. 601). I call it Rec. 601. To compute nonlinear video luma from nonlinear red, green and blue:

The prime symbols in this equation, and in those to follow, denote nonlinear components.

12 What are CIE XYZ components?

The CIE system is based on the description of color as a luminance component Y, as described above, and two additional components X and Z. The spectral weighting curves of X and Z have been standardized by the CIE based on statistics from experiments involving human observers. XYZ tristimulus values can describe any color. (RGB tristimulus values will be described later.)

The magnitudes of the XYZ components are proportional to physical energy, but their spectral composition corresponds to the color matching characteristics of human vision.

The CIE system is defined in Publication CIE No 15.2, Colorimetry, Second Edition (1986) [5].

13 Does my scanner use the CIE spectral curves?

Probably not. Scanners are most often used to scan images such as color photographs and color offset prints that are already “records” of three components of color information. The usual task of a scanner is not spec-tral analysis but extraction of the values of the three components that have already been recorded. Narrowband filters are more suited to this task than filters that adhere to the principles of colorimetry.

If you place on your scanner an original colored object that has “original” SPDs that are not already a record of three components, chances are your scanner will not very report accurate RGB values. This is because most scanners do not conform very closely to CIE standards.

14 What are CIE x and y chromaticity coordinates?

It is often convenient to discuss “pure” color in the absence of brightness. The CIE defines a normalization process to compute “little” x and y chro-maticity coordinates:

A color plots as a point in an (x, y) chromaticity diagram. When a narrow-band SPD comprising power at just one wavelength is swept across the range 400 nm to 700 nm, it traces a shark-fin shaped spectral locus in (x, y) coordinates. The sensation of purple cannot be produced by a single wavelength: to produce purple requires a mixture of shortwave and long-wave light. The line of purples on a chromaticity diagram joins extreme blue to extreme red. All colors are contained in the area in (x, y) bounded by the line of purples and the spectral locus.

′ = ′ + ′ + ′Y R G B601 0 299 0 587 0 114. . .

xX

X Y Zy

Y

X Y Z=

+ +=

+ +

8 Frequently Asked Questions About Color

A color can be specified by its chromaticity and luminance, in the form of an xyY triple. To recover X and Z from chromaticities and luminance, use these relations:

The bible of color science is Wyszecki and Styles, Color Science [6]. But it’s daunting. For Wyszecki’s own condensed version, see Color in Business, Science and Industry, Third Edition [7]. It is directed to the color industry: ink, paint and the like. For an approachable introduction to the same theory, accompanied by descriptions of image reproduction, try to find a copy of R.W.G. Hunt, The Reproduction of Colour [8]. But sorry to report, as I write this, it’s out of print.

15 What is white? In additive image reproduction, the white point is the chromaticity of the color reproduced by equal red, green and blue components. White point is a function of the ratio (or balance) of power among the primaries. In subtractive reproduction, white is the SPD of the illumination, multiplied by the SPD of the media. There is no unique physical or perceptual defini-tion of white, so to achieve accurate color interchange you must specify the characteristics of your white.

It is often convenient for purposes of calculation to define white as a uniform SPD. This white reference is known as the equal-energy illuminant, or CIE Illuminant E.

A more realistic reference that approximates daylight has been specified numerically by the CIE as Illuminant D65. You should use this unless you have a good reason to use something else. The print industry commonly uses D50 and photography commonly uses D55. These represent compro-mises between the conditions of indoor (tungsten) and daylight viewing.

16 What is color temperature?

Planck determined that the SPD radiated from a hot object – a black body radiator – is a function of the temperature to which the object is heated. Many sources of illumination have, at their core, a heated object, so it is often useful to characterize an illuminant by specifying the temperature (in units of kelvin, K) of a black body radiator that appears to have the same hue.

Although an illuminant can be specified informally by its color tempera-ture, a more complete specification is provided by the chromaticity coor-dinates of the SPD of the source.

Modern blue CRT phosphors are more efficient with respect to human vision than red or green. In a quest for brightness at the expense of color accuracy, it is common for a computer display to have excessive blue content, about twice as blue as daylight, with white at about 9300 K.

Human vision adapts to white in the viewing environment. An image viewed in isolation – such as a slide projected in a dark room – creates its own white reference, and a viewer will be quite tolerant of errors in the white point. But if the same image is viewed in the presence of an external white reference or a second image, then differences in white point can be objectionable.

Xx

yY Z

x y

yY= = − −1

24

Frequently Asked Questions About Color 9

Complete adaptation seems to be confined to the range 5000 K to 5500 K. For most people, D65 has a little hint of blue. Tungsten illumination, at about 3200 K, always appears somewhat yellow.

17 How can I characterize red, green and blue?

Additive reproduction is based on physical devices that produce all-posi-tive SPDs for each primary. Physically and mathematically, the spectra add. The largest range of colors will be produced with primaries that appear red, green and blue. Human color vision obeys the principle of superposition, so the color produced by any additive mixture of three primary spectra can be predicted by adding the corresponding fractions of the XYZ components of the primaries: the colors that can be mixed from a particular set of RGB primaries are completely determined by the colors of the primaries by themselves. Subtractive reproduction is much more complicated: the colors of mixtures are determined by the primaries and by the colors of their combinations.

An additive RGB system is specified by the chromaticities of its primaries and its white point. The extent (gamut) of the colors that can be mixed from a given set of RGB primaries is given in the (x, y) chromaticity diagram by a triangle whose vertices are the chromaticities of the prima-ries.

In computing there are no standard primaries or white point. If you have an RGB image but have no information about its chromaticities, you cannot accurately reproduce the image.

The NTSC in 1953 specified a set of primaries that were representative of phosphors used in color CRTs of that era. But phosphors changed over the years, primarily in response to market pressures for brighter receivers, and by the time of the first the videotape recorder the primaries in use were quite different than those “on the books”. So although you may see the NTSC primary chromaticities documented, they are of no use today.

Contemporary studio monitors have slightly different standards in North America, Europe and Japan. But international agreement has been obtained on primaries for high definition television (HDTV), and these primaries are closely representative of contemporary monitors in studio video, computing and computer graphics. The primaries and the D65 white point of Rec. 709 [9] are:

For a discussion of nonlinear RGB in computer graphics, see Lindbloom [10]. For technical details on monitor calibration, consult Cowan [11].

18 How do I transform between CIE XYZ and a particular set of RGB primaries?

RGB values in a particular set of primaries can be transformed to and from CIE XYZ by a three-by-three matrix transform. These transforms involve tristimulus values, that is, sets of three linear-light components that conform to the CIE color matching functions. CIE XYZ is a special case of tristimulus values. In XYZ, any color is represented by a positive set of values.

R G B whitex 0.640 0.300 0.150 0.3127y 0.330 0.600 0.060 0.3290z 0.030 0.100 0.790 0.3582

10 Frequently Asked Questions About Color

Details can be found in SMPTE RP 177-1993 [12].

To transform from CIE XYZ into Rec. 709 RGB (with its D65 white point), use this transform:

This matrix has some negative coefficients: XYZ colors that are out of gamut for a particular RGB transform to RGB where one or more RGB components is negative or greater than unity.

Here’s the inverse transform. Because white is normalized to unity, the middle row sums to unity:

To recover primary chromaticities from such a matrix, compute little x and y for each RGB column vector. To recover the white point, transform RGB=[1, 1, 1] to XYZ, then compute x and y.

19 Is RGB always device-dependent?

Video standards specify abstract R’G’B’ systems that are closely matched to the characteristics of real monitors. Physical devices that produce addi-tive color involve tolerances and uncertainties, but if you have a monitor that conforms to Rec. 709 within some tolerance, you can consider the monitor to be device-independent.

The importance of Rec. 709 as an interchange standard in studio video, broadcast television and high definition television, and the perceptual basis of the standard, assures that its parameters will be used even by devices such as flat-panel displays that do not have the same physics as CRTs.

20 How do I transform data from one set of RGB primaries to another?

RGB values in a system employing one set of primaries can be trans-formed into another set by a three-by-three linear-light matrix transform. Generally these matrices are normalized for a white point luminance of unity. For details, see Television Engineering Handbook [13].

As an example, here is the transform from SMPTE 240M (or SMPTE RP 145) RGB to Rec. 709:

All of these terms are close to either zero or one. In a case like this, if the transform is computed in the nonlinear (gamma-corrected) R’G’B’ domain the resulting errors will be insignificant.

R

G

B

X

Y

Z

709

709

709

3 240479 1 537150 0 498535

0 969256 1 875992 0 041556

0 055648 0 204043 1 057311

=− −

−−

•

. . .

. . .

. . .

X

Y

Z

R

G

B

=

•

0 412453 0 357580 0 180423

0 212671 0 715160 0 072169

0 019334 0 119193 0 950227

709

709

709

. . .

. . .

. . .

R

G

B

R

G

B

M

M

M

709

709

709

240

240

240

0 939555 0 050173 0 010272

0 017775 0 965795 0 016430

0 001622 0 004371 1 005993

=− −

•

. . .

. . .

. . .

24

Frequently Asked Questions About Color 11

Here’s another example. To transform EBU 3213 RGB to Rec. 709:

Transforming among RGB systems may lead to an out of gamut RGB result where one or more RGB components is negative or greater than unity.

21 Should I use RGB or XYZ for image synthesis?

Once light is on its way to the eye, any tristimulus-based system will work. But the interaction of light and objects involves spectra, not tristim-ulus values. In synthetic computer graphics, the calculations are actually simulating sampled SPDs, even if only three components are used. Details concerning the resultant errors are found in Hall [14].

22 What is subtractive color? Subtractive systems involve colored dyes or filters that absorb power from selected regions of the spectrum. The three filters are placed in tandem. A dye that appears cyan absobs longwave (red) light. By control-ling the amount of cyan dye (or ink), you modulate the amount of red in the image.

In physical terms the spectral transmission curves of the colorants multiply, so this method of color reproduction should really be called “multiplicative”. Photographers and printers have for decades measured transmission in base-10 logarithmic density units, where transmission of unity corresponds to a density of 0, transmission of 0.1 corresponds to a density of 1, transmission of 0.01 corresponds to a density of 2 and so on. When a printer or photographer computes the effect of filters in tandem, he subtracts density values instead of multiplying transmission values, so he calls the system subtractive.

To achieve a wide range of colors in a subtractive system requires filters that appear colored cyan, yellow and magenta (CMY). Cyan in tandem with magenta produces blue, cyan with yellow produces green, and magenta with yellow produces red. Smadar Nehab suggests this memory aid:

Additive primaries are at the top, subtractive at the bottom. On the left, magenta and yellow filters combine to produce red. On the right, red and green sources add to produce yellow.

23 Why did my grade three teacher tell me that the primaries are red, yellow and blue?

To get a wide range of colors in an additive system, the primaries must appear red, green and blue (RGB). In a subtractive system the primaries must appear yellow, cyan and magenta (CMY). It is complicated to predict the colors produced when mixing paints, but roughly speaking, paints mix additively to the extent that they are opaque (like oil paints), and subtractively to the extent that they are transparent (like water-colors). This question also relates to color names: your grade three “red” was probably a little on the magenta side, and “blue” was probably quite cyan. For a discussion of paint mixing from a computer graphics perspec-tive, consult Haase [15].

R

G

B

R

G

B

EBU

EBU

EBU

709

709

709

1 044036 0 044036 0

0 1 0

0 0 011797 0 988203

=−

•

. . .

. . .

. . .

R G B R G B

Cy Mg Yl Cy Mg Yl

12 Frequently Asked Questions About Color

24 Is CMY just one-minus-RGB?

In a theoretical subtractive system, CMY filters could have spectral absorption curves with no overlap. The color reproduction of the system would correspond exactly to additive color reproduction using the red, green and blue primaries that resulted from pairs of filters in combina-tion.

Practical photographic dyes and offset printing inks have spectral absorp-tion curves that overlap significantly. Most magenta dyes absorb medi-umwave (green) light as expected, but incidentally absorb about half that amount of shortwave (blue) light. If reproduction of a color, say brown, requires absorption of all shortwave light then the incidental absorption from the magenta dye is not noticed. But for other colors, the “one minus RGB” formula produces mixtures with much less blue than expected, and therefore produce pictures that have a yellow cast in the mid tones. Similar but less severe interactions are evident for the other pairs of prac-tical inks and dyes.

Due to the spectral overlap among the colorants, converting CMY using the “one-minus-RGB” method works for applications such as business graphics where accurate color need not be preserved, but the method fails to produce acceptable color images.

Multiplicative mixture in a CMY system is mathematically nonlinear, and the effect of the unwanted absorptions cannot be easily analyzed or compensated. The colors that can be mixed from a particular set of CMY primaries cannot be determined from the colors of the primaries them-selves, but are also a function of the colors of the sets of combinations of the primaries.

Print and photographic reproduction is also complicated by nonlineari-ties in the response of the three (or four) channels. In offset printing, the physical and optical processes of dot gain introduce nonlinearity that is roughly comparable to gamma correction in video. In a typical system used for print, a black code of 128 (on a scale of 0 to 255) produces a reflectance of about 0.26, not the 0.5 that you would expect from a linear system. Computations cannot be meaningfully performed on CMY components without taking nonlinearity into account.

For a detailed discussion of transferring colorimetric image data to print media, see Stone [16].

25 Why does offset printing use black ink in addition to CMY?

Printing black by overlaying cyan, yellow and magenta ink in offset printing has three major problems. First, colored ink is expensive. Replacing colored ink by black ink – which is primarily carbon – makes economic sense. Second, printing three ink layers causes the printed paper to become quite wet. If three inks can be replaced by one, the ink will dry more quickly, the press can be run faster, and the job will be less expensive. Third, if black is printed by combining three inks, and mechanical tolerances cause the three inks to be printed slightly out of register, then black edges will suffer colored tinges. Vision is most demanding of spatial detail in black and white areas. Printing black with a single ink minimizes the visibility of registration errors.

Other printing processes may or may not be subject to similar constraints.

24

Frequently Asked Questions About Color 13

26 What are color differences?

This term is ambiguous. In its first sense, color difference refers to numer-ical differences between color specifications. The perception of color differences in XYZ or RGB is highly nonuniform. The study of perceptual uniformity concerns numerical differences that correspond to color differ-ences at the threshold of perceptibility (just noticeable differences, or JNDs).

In its second sense, color difference refers to color components where brightness is “removed”. Vision has poor response to spatial detail in colored areas of the same luminance, compared to its response to lumi-nance spatial detail. If data capacity is at a premium it is advantageous to transmit luminance with full detail and to form two color difference components each having no contribution from luminance. The two color components can then have spatial detail removed by filtering, and can be transmitted with substantially less information capacity than luminance.

Instead of using a true luminance component to represent brightness, it is ubiquitous for practical reasons to use a luma signal that is computed nonlinearly as outlined above (What is luma? ).

The easiest way to “remove” brightness information to form two color channels is to subtract it. The luma component already contains a large fraction of the green information from the image, so it is standard to form the other two components by subtracting luma from nonlinear blue (to form B’-Y’ ) and by subtracting luma from nonlinear red (to form R’-Y’). These are called chroma.

Various scale factors are applied to (B’-Y’ ) and (R’-Y’) for different appli-cations. The Y ’PBPR scale factors are optimized for component analog video. The Y ’CBCR scaling is appropriate for component digital video such as studio video, JPEG and MPEG. Kodak’s PhotoYCC uses scale factors optimized for the gamut of film colors. Y’UV scaling is appropriate as an intermediate step in the formation of composite NTSC or PAL video signals, but is not appropriate when the components are kept separate. The Y’UV nomenclature is now used rather loosely, and it sometimes denotes any scaling of (B’-Y’ ) and (R’-Y’). Y ’IQ coding is obsolete.

The subscripts in CBCR and PBPR are often written in lower case. I find this to compromise readability, so without introducing any ambiguity I write them in uppercase. Authors with great attention to detail some-times “prime” these quantities to indicate their nonlinear nature, but because no practical image coding system employs linear color differ-ences I consider it safe to omit the primes.

14 Frequently Asked Questions About Color

27 How do I obtain color difference components from tristimulus values?

Here is the block diagram for luma/color difference encoding and decoding:

From linear XYZ – or linear R1G1B1 whose chromaticity coordinates are different from the interchange standard – apply a 3×3 matrix transform to obtain linear RGB according to the interchange primaries. Apply a nonlinear transfer function (“gamma correction”) to each of the compo-nents to get nonlinear R’G’B’ . Apply a 3×3 matrix to obtain color differ-ence components such as Y’PBPR, Y’CBCR or PhotoYCC. If necessary, apply a color subsampling filter to obtain subsampled color difference components. To decode, invert the above procedure: run through the block diagram right-to-left using the inverse operations. If your monitor conforms to the interchange primaries, decoding need not explicitly use a transfer function or the tristimulus 3×3.

The block diagram emphasizes that 3×3 matrix transforms are used for two distinctly different tasks. When someone hands you a 3×3, you have to ask for which task it is intended.

28 How do I encode Y'PBPR components?

Although the following matrices could in theory be used for tristimulus signals, it is ubiquitous to use them with gamma-corrected signals.

To encode Y’PBPR , start with the basic Y’, (B’-Y’ ) and (R’-Y’) relation-ships:

Eq 1

Y’PBPR components have unity excursion, where Y’ ranges [0..+1] and

each of PB and PR ranges [-0.5..+0.5]. The (B’-Y’ ) and (R’-Y’) rows need to

[M] [M]

RGBXYZor R1G1B1

R'G'B' Y'CBCR

Y'CBCRe.g. 4:2:2

TRISTIMULUS3 ×3

TRANSFERFUNCTION

COLOR DIFF.ENCODE

SUBSAMPLINGFILTER

[M] [M]

TRISTIMULUS3 ×3

TRANSFERFUNCTION

COLOR DIFF.DECODE

INTERPOLATIONFILTER

2.5

0.45

′′ − ′′ − ′

= − −− −

•′′′

Y

B Y

R Y

R

G

B

601

601

601

0 299 0 587 0 114

0 299 0 587 0 886

0 701 0 587 0 114

. . .

. . .

. . .

24

Frequently Asked Questions About Color 15

be scaled by and . To encode from R’G’B’ where reference

black is 0 and reference white is +1:

Eq 2

The first row comprises the luma coefficients; these sum to unity. The second and third rows each sum to zero, a necessity for color difference components. The +0.5 entries reflect the maximum excursion of PB and PR of +0.5, for the blue and red primaries [0, 0, 1] and [1, 0, 0].

The inverse, decoding matrix is this:

29 How do I encode Y'CBCR components from R'G'B' in [0, +1]?

Rec. 601 specifies eight-bit coding where Y’ has an excursion of 219 and an offset of +16. This coding places black at code 16 and white at code 235, reserving the extremes of the range for signal processing headroom and footroom. CB and CR have excursions of ±112 and offset of +128, for a range of 16 through 240 inclusive.

To compute Y’CBCR from R’G’B’ in the range [0..+1], scale the rows of the matrix of Eq 2 by the factors 219, 224 and 224, corresponding to the excur-sions of each of the components:

Eq 3

Summing the first row of the matrix yields 219, the luma excursion from black to white. The two entries of 112 reflect the positive CBCR extrema of the blue and red primaries.

Clamp all three components to the range 1 through 254 inclusive, since Rec. 601 reserves codes 0 and 255 for synchronization signals.

To recover R’G’B’ in the range [0..+1] from Y’CBCR, use the inverse of Eq 3 above:

This looks overwhelming, but the Y’CBCR components are integers in eight bits and the reconstructed R’G’B’ are scaled down to the range [0..+1].

30 How do I encode Y'CBCR components from computer R'G'B' ?

In computing it is conventional to use eight-bit coding with black at code 0 and white at 255. To encode Y’CBCR from R’G’B’ in the range [0..255],

0.50.886------------- 0.5

0.701-------------

′

= − −− −

•′′′

Y

P

P

R

G

BB

R

601 0 299 0 587 0 114

0 168736 0 331264 0 5

0 5 0 418688 0 081312

. . .

. . .

. . .

′′′

= − −

•′

R

G

B

Y

P

PB

R

1 0 1 402

1 0 344136 0 714136

1 1 772 0

601.

. .

.

′

=

+ − −− −

•′′′

Y

C

C

R

G

BB

R

601 16

128

128

65 481 128 553 24 966

37 797 74 203 112

112 93 786 18 214

. . .

. . .

. . .

′′′

= − −

•′

−

R

G

B

Y

C

CB

R

0 00456621 0 0 00625893

0 00456621 0 00153632 0 00318811

0 00456621 0 00791071 0

16

128

128

601. . .

. . .

. . .

16 Frequently Asked Questions About Color

using eight-bit binary arithmetic, scale the Y’CBCR matrix of Eq 3 by 256⁄ 255:

To decode R’G’B’ in the range [0..255] from Rec. 601 Y’CBCR, using eight-bit binary arithmetic:

Eq 4

The multiplications by 1⁄ 256 can be accomplished by shifting. Some of the coefficients, when scaled by 1⁄ 256, are larger than unity. These coefficients will need more than eight multiplier bits.

For implementation in binary arithmetic the matrix coefficients have to be rounded. When you round, take care to preserve the row sums of [1, 0, 0].

The matrix of Eq 4 will decode standard Y’CBCR components to RGB components in the range [0..255], subject to roundoff error. You must take care to avoid overflow due to roundoff error. But you must protect against overflow in any case, because studio video signals use the extremes of the coding range to handle signal overshoot and undershoot, and these will require clipping when decoded to an RGB range that has no headroom or footroom.

31 How do I encode Y'CBCR components from studio video?

Studio R’G’B’ signals use the same 219 excursion as the luma component of Y’CBCR.To encode Y’CBCR from R’G’B’ in the range [0..219], using eight-bit binary arithmetic, scale the Y’CBCR encoding matrix of Eq 3 above by 256⁄ 219. Here is the encoding transform for studio video:

To decode R’G’B’ in the range [0..219] from Y’CBCR, using eight-bit binary arithmetic:

The entries of 256 in this matrix indicate that the corresponding compo-nent can simply be added; there is no need for a multiplication operation. This matrix contains entries larger than 256; the corresponding multi-pliers will need capability for nine bits.

The matrices in this section conform to Rec. 601 and apply directly to conventional 525/59.94 and 625/50 video. It is not yet decided whether emerging HDTV standards will use the same matrices, or adopt a new set of matrices having different luma coefficients. In my view it would be

′

=

+ − −− −

•′′′

Y

C

C

R

G

BB

R

601 255

255

255

16

128

128

1256

65 738 129 057 25 064

37 945 74 494 112 439

112 439 94 154 18 285

. . .

. . .

. . .

′′′

= − −

•′

−

R

G

B

Y

C

CB

R

255

255

255

6011

256

298 082 0 408 583

298 082 100 291 208 120

298 082 516 411 0

16

128

128

. . .

. . .

. . .

′

=

+ − −− −

•′′′

Y

C

C

R

G

BB

R

601 219

219

219

16

128

128

1256

76 544 150 272 29 184

44 182 86 740 130 922

130 922 109 631 21 291

. . .

. . .

. . .

′′′

= − −

•′

−

R

G

B

Y

C

CB

R

219

219

219

6011

256

256 0 350 901

256 86 132 178 738

256 443 506 0

16

128

128

. . .

. . .

. . .

24

Frequently Asked Questions About Color 17

unfortunate if different matrices were adopted, because then image coding and decoding would depend on whether the picture was small (conventional video) or large (HDTV).

In digital video, Rec. 601 standardizes subsampling denoted 4:2:2, where CB and CR components are subsampled horizontally by a factor of two with respect to luma. JPEG and MPEG conventionally subsample by a factor of two in the vertical dimension as well, denoted 4:2:0.

Color difference coding is standardized in Rec. 601. For details on color difference coding as used in video, consult Poynton [17].

32 How do I decode R'G'B' from PhotoYCC ?

Kodak’s PhotoYCC uses the Rec. 709 primaries, white point and transfer function. Reference white codes to luma 189; this preserves film high-lights. The color difference coding is asymmetrical, to encompass film gamut. You are unlikely to encounter any raw image data in PhotoYCC form because YCC is closely associated with the PhotoCD system whose compression methods are proprietary. But just in case, the following equation is comparable to in that it produces R’G’B’ in the range [0..+1] from integer YCC. If you want to return R’G’B’ in a different range, or implement the equation in eight-bit integer arithmetic, use the techniques in the section above.

Decoded R’G’B’ components from PhotoYCC can exceed unity or go below zero. PhotoYCC extends the Rec. 709 transfer function above unity, and reflects it around zero, to accommodate wide excursions of R’G’B’ . To decode to CRT primaries, clip R’G’B’ to the range zero to one.

33 Will you tell me how to decode Y'UV and Y'IQ?

No, I won’t! Y’UV and Y’IQ have scale factors appropriate to composite NTSC and PAL. They have no place in component digital video! You shouldn’t code into these systems, and if someone hands you an image claiming it’s Y’UV, chances are it’s actually Y’CBCR, it’s got the wrong scale factors, or it’s linear-light.

Well OK, just this once. To transform Y’, (B’-Y’ ) and (R’-Y’) components from Eq 1 to Y’UV, scale (B’-Y’) by 0.492111 to get U and scale R’-Y’ by 0.877283 to get V. The factors are chosen to limit composite NTSC or PAL amplitude for all legal R’G’B’ values:

To transform from Y’IQ to Y’UV, perform a 33° rotation and an exchange of color difference axes:

34 How should I test my encoders and decoders?

To test your encoding and decoding, ensure that colorbars are handled correctly. A colorbar signal comprises a binary RGB sequence ordered for

′′′

= − −

•′

−

R

G

B

Y

C

C

709

709

709

601189

1

2

0 0054980 0 0 0051681

0 0054980 0 0015446 0 0026325

0 0054980 0 0079533 0

0

156

137

. . .

. . .

. . .

,

− ≤ ′ ± ′ − ′( ) + ′ − ′( )[ ] ≤13

0 492111 0 87728343

Y B Y R Y. .

′

= −

•′

Y

U

V

Y

I

Q

601 6011 0 0

0 0 544639 0 838671

0 0 838671 0 544639

. .

. .

18 Frequently Asked Questions About Color

decreasing luma: white, yellow, cyan, green, magenta, red, blue and black.

To ensure that your scale factors are correct and that clipping is not being invoked, test 75% bars, a colorbar sequence having 75%-amplitude bars instead of 100%.

35 What is perceptual uniformity?

A system is perceptually uniform if a small perturbation to a component value is approximately equally perceptible across the range of that value. The volume control on your radio is designed to be perceptually uniform: rotating the knob ten degrees produces approximately the same percep-tual increment in volume anywhere across the range of the control. If the control were physically linear, the logarithmic nature of human loudness perception would place all of the perceptual “action” of the control at the bottom of its range.

The XYZ and RGB systems are far from exhibiting perceptual uniformity. Finding a transformation of XYZ into a reasonably perceptually-uniform space consumed a decade or more at the CIE and in the end no single system could be agreed. So the CIE standardized two systems, L*u*v* and L*a*b*, sometimes written CIELUV and CIELAB. (The u and v are unre-lated to video U and V.) Both L*u*v* and L*a*b* improve the 80:1 or so perceptual nonuniformity of XYZ to about 6:1. Both demand too much computation to accommodate real-time display, although both have been successfully applied to image coding for printing.

Computation of CIE L*u*v* involves intermediate u’ and v’ quantities, where the prime denotes the successor to the obsolete 1960 CIE u and v system:

First compute un’ and vn’ for your reference white Xn, Yn and Zn. Then compute u’ and v‘ – and L* as discussed earlier – for your colors. Finally, compute:

L*a*b* is computed as follows, for (X/Xn, Y/Yn, Z/Zn) > 0.01:

These equations are great for a few spot colors, but no fun for a million pixels. Although it was not specifically optimized for this purpose, the nonlinear R’G’B’ coding used in video is quite perceptually uniform, and has the advantage of being fast enough for interactive applications.

1 1 0 0 1 1 0 0

1 1 1 1 0 0 0 0

1 0 1 0 1 0 1 0

uX

X Y Zv

Y

X Y Z' , '=

+ +=

+ +4

15 39

15 3

u L u u v L v vn n* * * *,= ′ − ′( ) = ′ − ′( )13 13

aX

X

Y

Yb

Y

Y

Z

Zn n n n

* *,=

−

=

−

500 200

13

13

13

13

24

Frequently Asked Questions About Color 19

36 What are HSB and HLS? HSB and HLS were developed to specify numerical Hue, Saturation and Brightness (or Hue, Lightness and Saturation) in an age when users had to specify colors numerically. The usual formulations of HSB and HLS are flawed with respect to the properties of color vision. Now that users can choose colors visually, or choose colors related to other media (such as PANTONE), or use perceptually-based systems like L*u*v* and L*a*b*, HSB and HLS should be abandoned.

Here are some of problems of HSB and HLS. In color selection where “lightness” runs from zero to 100, a lightness of 50 should appear to be half as bright as a lightness of 100. But the usual formulations of HSB and HLS make no reference to the linearity or nonlinearity of the underlying RGB, and make no reference to the lightness perception of human vision.

The usual formulation of HSB and HLS compute so-called “lightness” or “brightness” as (R+G+B)/3. This computation conflicts badly with the properties of color vision, as it computes yellow to be about six times more intense than blue with the same “lightness” value (say L=50).

HSB and HSL are not useful for image computation because of the discontinuity of hue at 360°. You cannot perform arithmetic mixtures of colors expressed in polar coordinates.

Nearly all formulations of HSB and HLS involve different computations around 60° segments of the hue circle. These calculations introduce visible discontinuities in color space.

Although the claim is made that HSB and HLS are “device independent”, the ubiquitous formulations are based on RGB components whose chro-maticities and white point are unspecified. Consequently, HSB and HLS are useless for conveyance of accurate color information.

If you really need to specify hue and saturation by numerical values, rather than HSB and HSL you should use polar coordinate version of u* and v*: h*uv for hue angle and c*uvfor chroma.

37 What is true color? True color is the provision of three separate components for additive red, green and blue reproduction. A high quality true color system provides 8 bits for each of the three components; this is known as 24 bit color.

A high-quality true color system interposes a lookup table between each component of the framestore and each channel of the display. This makes it possible to use a true color system with either linear or nonlinear coding. In the X Window System, true color refers to fixed lookup tables, and direct color refers to lookup tables that are under the control of appli-cation software.

A hicolor system provides 16 bits for each pixel, partitioned into red, green, and blue components. Hicolor is a variant of truecolor, but with an insufficient number of bits to provide photographic quality. The 16 bits may be partitioned as 5 bits for each component (with the extra bit some-times used to convey transparency), or as 5 bits of red, 6 bits of green, and 5 bits of blue. Hicolor systems usually offer no lookup table at the output of the framebuffer, so the image data is coded like video: The RGB components are assumed to have been raised to a power of about 0.45.

20 Frequently Asked Questions About Color

38 What is indexed color? Indexed color (or pseudocolor), is the provision of a relatively small number of discrete colors – often 256 – in a colormap or palette. The framebuffer stores, at each pixel, the index number of a color. At the output of the framebuffer, a lookup table uses the index to retrieve red, green and blue components that are then sent to the display.

The colors in the map may be fixed systematically at the design of a system. As an example, 216 index entries an eight-bit indexed color system can be partitioned systematically into a 6×6×6 colorcube to imple-ment what amounts to a direct color system where each of red, green and blue has a value that is an integer in the range zero to five.

An RGB image can be converted to a predetermined colormap by choosing, for each pixel in the image, the colormap index corresponding to the “closest” RGB triple. With a systematic colormap such as a 6×6×6 colorcube this is straightforward. For an arbitrary colormap, the colormap has to be searched looking for entries that are “close” to the requested color. “Closeness” should be determined according to the perceptibility of color differences. Using color systems such as CIE L*u*v* or L*a*b* is computationally prohibitive, but in practice it is adequate to use a Euclidean distance metric in R’G’B’ components coded nonlinearly according to video practice.

A direct color image can be converted to indexed color with an image-dependent colormap by a process of color quantization that searches through all of the triples used in the image, and chooses the palette for the image based on the colors that are in some sense most “important”. Again, the decisions should be made according to the perceptibility of color differences. Adobe Photoshop can perform this conversion. UNIX users can employ the pbm package.

If your system accommodates arbitrary colormaps, when the map associ-ated with the image in a particular window is loaded into the hardware colormap, the maps associated with other windows may be disturbed. In window system such as the X Window System running on a multi-tasking operating system such as UNIX, even moving the cursor between two windows with different maps can cause annoying colormap flashing.

An eight-bit indexed color system requires less data to represent a picture than a twenty-four bit truecolor system. But this data reduction comes at a high price. The truecolor system can represent each of its three compo-nents according to the principles of sampled continuous signals. This makes it possible to accomplish, with good quality, operations such as resizing the image. In indexed color these operations introduce severe artifacts because the underlying representation lacks the properties of a continuous representation, even if converted back to RGB.

In graphic file formats such as GIF of TIFF, an indexed color image is accompanied by its colormap. Generally such a colormap has RGB entries that are gamma corrected: the colormap’s RGB codes are intended to be presented directly to a CRT, without further gamma correction.

24

Frequently Asked Questions About Color 21

39 I want to visualize a scalar function of two variables. Should I use RGB values corresponding to the colors of the rainbow?

When you look at a rainbow you do not see a smooth gradation of colors. Instead, some bands appear quite narrow, and others are quite broad. Perceptibility of hue variation near 540 nm is half that of either 500 nm or 600 nm. If you use the rainbow’s colors to represent data, the visibility of differences among your data values will depend on where they lie in the spectrum.

If you are using color to aid in the visual detection of patterns, you should use colors chosen according to the principles of perceptual uniformity. This an open research problem, but basing your system on CIE L*a*b* or L*u*v*, or on nonlinear video-like RGB, would be a good start.

40 What is dithering? A display device may have only a small number of choices of greyscale values or color values at each device pixel. However if the viewer is suffi-ciently distant from the display, the value of neighboring pixels can be set so that the viewer’s eye integrates several pixels to achieve an apparent improvement in the number of levels or colors that can be reproduced.

Computer displays are generally viewed from distances where the device pixels subtend a rather large angle at the viewer’s eye, relative to his visual acuity. Applying dither to a conventional computer display often introduces objectionable artifacts. However, careful application of dither can be effective. For example, human vision has poor acuity for blue spatial detail but good color discrimination capability in blue. Blue can be dithered across two-by-two pixel arrays to produce four times the number of blue levels, with no perceptible penalty at normal viewing distances.

41 How does halftoning relate to color?

The processes of offset printing and conventional laser printing are intrin-sically bilevel: a particular location on the page is either covered with ink or not. However, each of these devices can reproduce closely-spaced dots of variable size. An array of small dots produces the perception of light gray, and an array of large dots produces dark gray. This process is called halftoning or screening. In a sense this is dithering, but with device dots so small that acceptable pictures can be produced at reasonable viewing distances.

Halftone dots are usually placed in a regular grid, although stochastic screening has recently been introduced that modulates the spacing of the dots rather than their size.

In color printing it is conventional to use cyan, magenta, yellow and black grids that have exactly the same dot pitch but different carefully-chosen screen angles. The recently introduced technique of Flamenco screening uses the same screen angles for all screens, but its registration require-ments are more stringent than conventional offset printing.

Agfa’s booklet [18] is an excellent introduction to practical concerns of printing. And it’s in color! The standard reference to halftoning algo-rithms is Ulichney [19], but that work does not detail the nonlinearities found in practical printing systems. For details about screening for color reproduction, consult Fink [20]. Consult Frequently Asked Questions about Gamma for an introduction to the transfer function of offset printing.

22 Frequently Asked Questions About Color

42 What’s a color management system?

Software and hardware for scanner, monitor and printer calibration have had limited success in dealing with the inaccuracies of color handling in desktop computing. These solutions deal with specific pairs of devices but cannot address the end-to-end system. Certain application devel-opers have added color transformation capability to their applications, but the majority of application developers have insufficient expertise and insufficient resources to invest in accurate color.

A color management system (CMS) is a layer of software resident on a computer that negotiates color reproduction between the application and color devices. It cooperates with the operating system and the graphics library components of the platform software. Color management systems perform the color transformations necessary to exchange accurate color between diverse devices, in various color coding systems including RGB, CMYK and CIE L*a*b*.

The CMS makes available to the application a set of facilities whereby the application can determine what color devices and what color spaces are available. When the application wishes to access a particular device, it requests that the color manager perform a mathematical transform from one space to another. The color spaces involved can be device-indepen-dent abstract color spaces such as CIE XYZ, CIE L*a*b* or calibrated RGB. Alternatively a color space can be associated with a particular device. In the second case the color manager needs access to characterization data for the device, and perhaps also to calibration data that reflects the state of the particular instance of the device.

Apple’s ColorSync provides an interface between a Mac application program and color management capabilities either built-in to ColorSync or provided by a plug-in. Kodak’s CMS is built-into the latest version of Sun’s Solaris operating system.

The basic CMS services provided with desktop operating systems are likely to be adequate for office users, but are unlikely to satisfy high-end users such as in prepress. All of the announced systems have provisions for plug-in color management modules (CMMs) that can provide sophisti-cated transform machinery. Advanced color management modules are commercially available from Kodak, Agfa, and others. For an application developer’s prespective on color management, see Aldus [21].

43 How does a CMS know about particular devices?

A CMS needs access to information that characterizes the color repro-duction capabilities of particular devices. The set of characterization data for a device is called a device profile. Industry agreement has been reached on the format of device profiles; information is available from the Interna-tional Color Consortium (ICC). Vendors of color peripherals will soon provide industry-standard profiles with their devices, and they will have to make, buy or rent characterization services.

If you have a device that has not been characterized by its manufacturer, Agfa’s FotoTune software – part of Agfa’s FotoFlow color manager – can create device profiles.

44 Is a color management system useful for color specification?

Not quite yet. But future color management system are likely to include the ability to accommodate commercial proprietary color specification systems such as PANTONE and COLORCURVE . These vendors are likely

24

Frequently Asked Questions About Color 23

to provide their color specification systems in shrink-wrapped form to plug into color managers. In this way, users will have guaranteed color accuracy among applications and peripherals, and application vendors will no longer need to pay to license these systems individually.

45 I’m not a color expert. What parameters should I use to code my images?

Use the CIE D65 white point (6504 K) if you can.

Use the Rec. 709 primary chromaticities. Your monitor is probably already quite close to this. Rec. 709 has international agreement, offers excellent performance, and is the basis for HDTV development so it’s future-proof.

If you need to operate in linear light, so be it. Otherwise, for best percep-tual performance and maximum ease of interchange with digital video, use the Rec. 709 transfer function, with its 0.45-power law. If you need Mac compatibility you will have to suffer a penalty in perceptual perfor-mance. Raise tristimulus values to the 1 ⁄1.4-power before presenting them to QuickDraw.

To code luma, use the Rec. 601 luma coefficients 0.299, 0.587 and 0.114. Use Rec. 601 digital video coding with black at 16 and white at 235.

Use prime symbols (’) to denote all of your nonlinear components!

PhotoCD uses all of the preceding measures. PhotoCD codes color differ-ences asymmetrically, according to film gamut. Unless you have a requirement for film gamut, you should code into color differences using Y‘CBCR coding with Rec. 601 studio video (16..235/128±112) excursion.

Tag your image data with the primary and white chromaticity, transfer function and luma coefficients that you are using. TIFF 6.0 tags have been defined for these parameters. This will enable intelligent readers, today or in the future, to determine the parameters of your coded image and give you the best possible results.

46 References [1] B. Berlin and P. Kay, Basic Color Terms (Berkeley, Calif.: U. of Calif. Press, 1969)

[2] Publication CIE No 17.4, International Lighting Vocabulary (Vienna, Austria: Central Bureau of the Commission Internationale de L’Éclairage)

[3] LeRoy E. DeMarsh and Edward J. Giorgianni, “Color Science for Imaging Systems,” in Physics Today, September 1989, 44-52.

[4] W.F. Schreiber, Fundamentals of Electronic Imaging Systems, Second Edition (Springer-Verlag, 1991)

[5] Publication CIE No 15.2, Colorimetry, Second Edition (Vienna, Austria: Central Bureau of the Commission Internationale de L’Éclairage, 1986)

[6] Günter Wyszecki and W.S. Styles, Color Science: Concepts and Methods, Quantitative Data and Formulae, Second Edition (New York: John Wiley & Sons, 1982)

[7] D.B. Judd and Günter Wyszecki, Color in Business, Science and Industry, Third Edition (New York: John Wiley & Sons, 1975)

[8] R.W.G. Hunt, The Reproduction of Colour in Photography, Printing and Tele-vision, Fourth Edition (Tolworth, England: Fountain Press, 1987)

24 Frequently Asked Questions About Color

[9] ITU-R Recommendation BT.709, Basic Parameter Values for the HDTV Stan-dard for the Studio and for International Programme Exchange (1990), [formerly CCIR Rec. 709] (Geneva: ITU, 1990)

[10] Bruce J. Lindbloom, “Accurate Color Reproduction for Computer Graphics Applications”, Computer Graphics, Vol. 23, No. 3 (July 1989), 117-126 (proceedings of SIGGRAPH ’89)

[11] William B. Cowan, “An Inexpensive Scheme for Calibration of a Colour Monitor in terms of CIE Standard Coordinates”, in Computer Graphics, Vol. 17, No. 3 (July 1983), 315-321.

[12] SMPTE RP 177-1993, Derivation of Basic Television Color Equations.

[13] Television Engineering Handbook, Featuring HDTV Systems, Revised Edition by K. Blair Benson, revised by Jerry C. Whitaker (New York: McGraw-Hill, 1992). This supersedes the Second Edition.

[14] Roy Hall, Illumination and Color in Computer Generated Imagery (Springer-Verlag, 1989)

[15] Chet S. Haase and Gary W. Meyer, “Modelling Pigmented Materials for Realistic Image Synthesis”, in ACM Transactions on Graphics, Vol. 11, No. 4, 1992, p. 305.

[16] Maureen C. Stone, William B. Cowan and John C. Beatty, “Color Gamut Mapping and the Printing of Digital Color Images”, in ACM Transactions on Graphics, Vol. 7, No. 3, October 1988.

[17] Charles Poynton, A Technical Introduction to Digital Video (New York: John Wiley & Sons, 1996)

[18] Agfa Corporation, An introduction to Digital Color Prepress, Volumes 1 and 2 (Mt.Prospect, Ill.: Prepress Education Resources, 800 395 7007, 1990)

[19] Robert Ulichney, Digital Halftoning (Cambridge, Mass.: MIT Press, 1988)

[20] Peter Fink, PostScript Screening: Adobe Accurate Screens (Mountain View, Calif.: Adobe Press, 1992)

[21] Color management systems: Getting reliable color from start to finish, Adobe Systems, <http://www.adobe.com/PDFs/FaxYI/500301.pdf>.

[22] Overview of color publishing, Adobe Systems, <http://www.adobe.com/PDFs/FaxYI/500302.pdf>. Despite appear-ances and title, this document is in greyscale, not color.

47 Contributors Thanks to Norbert Gerfelder, Alan Roberts and Fred Remley for their proofreading and editing. I learned about color from LeRoy DeMarsh, Ed Giorgianni, Junji Kumada and Bill Cowan. Thanks!

This note contains some errors: if you find any, please let me know.

I welcome suggestions for additions and improvements.

24

Home > Articles

The Visual System and the Brain: Hubel and Wiesel Redux

• By Dale Purves• Dec 21, 2009

This chapter is from the bookBrains: How They Seem to Work

Dale Purves explains the human visual system and how the work of David Hubel and Torsten Wiesel provided the greatest single influence on the ways neuroscientists think about the brain during much of the second half of the twentieth century.

I don't think many neuroscientists would dispute the statement that the work David Hubel and Torsten Wiesel began in the late 1950s and continued for the next 25 years provided the greatest single influence on the ways neuroscientists thought about and prosecuted studies of the brain during much of the second half of the twentieth century. Certainly, what they were doing had never been very far from my own thinking, even while working on the formation and maintenance of synaptic connections in the peripheral nervous system. To explain the impact of their work and to set the stage for understanding the issues discussed in the remaining chapters, I need to fill in more information about the visual system, what Hubel and Wiesel actually did, and how they interpreted it.

Presumably because we humans depend so heavily on vision, this sensory modality has for centuries been a focus of interest for natural philosophers and, in the modern era, neuroscientists and psychologists. By the time Hubel and Wiesel got into the game in the 1950s, a great deal was already known about the anatomy of the system and about the way light interacts with receptor cells in the retina to initiate the action potentials that travel centrally from retina to cortex, ultimately leading to what we see. The so-called primary visual pathway (Figure 7.1) begins with the two types of retinal receptors, rods and cones, and their transduction of light energy.

Figure 7.1 The primary visual pathway carries information from the eye to the regions of the brain that determine what we see. The pathway entails the retinas, optic nerves, optic tracts, dorsal lateral geniculate nuclei in the thalamus, optic radiations, and primary (or striate) and adjacent secondary (or extrastriate) visual cortices in each occipital lobe at the back of the brain (see Figures 7.2 and 7.3). Other central pathways to targets in the brainstem (dotted lines) determine pupil diameter as a function of retinal light levels, organize and motivate eye movements, and influence circadian rhythms. (After Purves and Lotto, 2003)

The visual processing that rods initiate is primarily concerned with seeing at very low light levels, whereas

cones respond only to greater light intensities and are responsible for the detail and color qualities that we normally think of as defining visual perception. However, the primary visual pathway is anything but simple. Following the extensive neural processing that takes place among the five basic cell classes found in the retina, information arising from both rods and cones converges onto the retinal ganglion cells, the neurons whose axons leave the retina in the optic nerve. The major targets of the retinal ganglion cells are the neurons in the dorsal lateral geniculate nucleus of the thalamus, which project to the primary visual cortex (usually referred to as V1 or the striate cortex) (Figure 7.2).

Figure 7.2 Photomicrograph of a section of the human primary visual cortex, taken in the plane of the face (see Figure 7.1). The characteristic myelinated band, or stria, is why this region of cortex is referred to as the striate cortex (myelin is a fatty material that invests most axons in the brain and so stains darkly with reagents that dissolve in fat, such as the one used). The primary visual cortex occupies about 25 square centimeters (about a third of the surface area of a dollar bill) in each cerebral hemisphere; the overall area of the cortical surface for the two hemispheres together is about 0.8 square meters (or, as my colleague Len White likes to tell students, about the area of a medium pizza). Most of the primary visual cortex lies within a fissure on the medial surface of the occipital lobe called the calcarine sulcus, which is also shown in Figure 6.1B. The extrastiate cortex that carries out further processing of visual information is immediately adjacent (see Figure 7.3). (Courtesy of T. Andrews and D. Purves)

Although the primary visual cortex (V1) is the nominal terminus of this pathway, many of the neurons there project to additional areas in the occipital, parietal, and temporal lobes (Figure 7.3). Neurons in V1 also interact extensively with each other and send information back to the thalamus, where much processing occurs that remains poorly understand. Because of the increasing integration of information from other brain regions in the visual cortical regions adjacent to V1, these higher-order cortical processing regions (V2, V3, and so on) are called visual association areas. Taken together, they are also referred to as extrastriate visual cortical areas because they lack the anatomically distinct layer that creates the striped appearance of V1 (see Figure 7.2). In most conceptions of vision, perception is thought to occur in these higher-order visual areas adjacent to V1 (although note that what occur means in this statement is not straightforward).

Figure 7.3 The higher-order visual cortical areas adjacent to the primary visual cortex, shown here in lateral (A) and medial (B) views of the brain. The primary visual cortex (V1) is indicated in green; the additional colored areas with their numbered names are together called the extrastriate or visual association areas and occupy much of the rest of the occipital lobe at the back of the brain (its anterior border is indicated by the dotted line).(After Purves and Lotto, 2003)

By the 1950s, much had also been learned about visual perception. The seminal figures in this aspect of the history of vision science were nineteenth-century German physicist and physiologist Hermann von Helmholtz, and Wilhelm Wundt and Gustav Fechner, who initiated the modern study of perception from a psychological perspective at about the same time. However, Helmholtz gave impetus to the effort to understand perception in terms of visual system physiology, and his work was the forerunner of the program Hubel and Wiesel

undertook nearly a century later.

A good example of Helmholtz's approach is his work on color vision. At the beginning of the nineteenth century, British natural philosopher Thomas Young had surmised that three distinct types of receptors in the human retina generate the perception of color. Although Young knew nothing about cones or the pigments in them that underlie light absorption, he nevertheless contended in lectures he gave to the Royal Society in 1802 that three different classes of receptive "particles" must exist. Young's argument was based on what humans perceive when lights of different wavelengths (loosely speaking, lights of different colors) are mixed, a methodology that had been used since Isaac Newton's discovery a hundred years earlier that light comprises a range of wavelengths. Young's key observation was that most color sensations can be produced by mixing appropriate amounts of lights from the long-, middle-, and short-wavelength regions of the visible light spectrum (mixing lights is called color addition and is different from mixing pigments, which subtracts particular wavelengths from the stimulus that reaches the eye by absorbing them).

Young's theory was largely ignored until the latter part of the nineteenth century, when it was revived and greatly extended by Helmholtz and James Clerk Maxwell, another highly accomplished physicist interested in vision. The ultimately correct idea that humans have three types of cones with sensitivities (absorption spectra) that peak in the long, middle, and short wavelength ranges, respectively, is referred to as trichromacy, denoting the fact that most human color sensations can be elicited in normal observers by adjusting the relative activation of the three cone types (see Chapter 9). The further hypothesis that the relative activation explains the colors we actually see is called the trichromacy theory, and Helmholtz spotlighted this approach to explaining perception. Helmholtz's approach implied that perceptions (color perceptions, in this instance) are a direct consequence of the way receptors and the higher-order neurons related to them analyze and ultimately represent stimulus features and, therefore, the features of objects in the world. For Helmholtz and many others since that era, the feature that color perceptions represent is the nature of object surfaces conveyed by the spectrum of light they reflect to the eye.

This mindset that sensory systems represent the features of objects in the world was certainly the way I had supposed the sensory components of the brain to be working—and as far as I could tell, it was how pretty much everyone else thought about these issues in the 1960s and 1970s. By the same token, I took exploring the underlying neural circuitry (the work Hubel and Wiesel were undertaking) to be the obvious way to solve the problem of how the visual system generates what we see. The step remaining was the hard work needed to determine how the physiology of individual visual neurons and their connections in the various stations of the visual pathway were accomplishing this feat.

Using the extracellular recording method they had developed in Kuffler's lab at Johns Hopkins, Hubel and Wiesel were working their way up the primary visual pathway in cats and, later, in monkeys. At each stage in the pathway—the thalamus, primary visual cortex, and, ultimately, extratstriate cortical areas (see Figures 7.1–7.3)—they carefully studied the response characteristics of individual neurons in the type of setup that Figure 7.4 illustrates, describing the results in terms of what are called the receptive field properties of visual neurons. Their initial studies of neurons in the lateral geniculate nucleus of the thalamus showed responses that were similar to the responses of the retinal output neurons (retinal ganglion cells) that Kuffler had described. Despite this similarity, the information the axons carried from the thalamus to the cortex was not exactly the same as the information coming into the nucleus from the retina, indicating some processing by the thalamus. The major advances, however, came during the next few years as they studied the responses of nerve cells in the primary visual cortex. The key finding was that, unlike the relatively nondescript responses to light stimuli of visual neurons in the retina or the thalamus, cortical neurons showed far more varied and specific responses. On the surface, the nature of these responses seemed closely related to the features we end up seeing. For example, the rather typical V1 neuron illustrated in Figure 7.4 responds to light stimuli presented at only one relatively small locus on the screen (defining the spatial limits of the neuron's receptive field), and only to bars of light. In contrast, neurons in the retina or thalamus respond to any configuration of light that falls within their receptive field. Moreover, many V1 neurons are selective for orientation and direction of movement, responding vigorously to bars only at or near a particular angle on the screen and moving in a particular direction. These receptive field properties were the beginning of what has eventually become a long list, including selective responses to the lengths of lines, different colors, input from one eye or the other, and the different depths indicated by the somewhat different views of the two eyes. Based on this rapidly accumulating evidence, it seemed clear that visual cortical neurons were indeed encoding the features of retinal images and, therefore, the

properties of objects in the world.

Figure 7.4 Assessing the responses of individual neurons to visual stimuli in experimental animals (although the animal is anesthetized, the visual system continues to operate much as it would if the animal were awake). A) Diagram of the experimental setup showing an extracellular electrode recording from a neuron in the primary visual cortex of a cat (which is more anterior in the brain than in humans). By monitoring the responses of the neuron to stimuli shown on a screen, Hubel and Wiesel could get a good idea of what particular visual neurons normally do. B) In this example, the neuron being recorded from in V1 responds selectively to bars of light presented on the screen in different orientations; the cell fires action potentials (indicated by the vertical lines) only when the bar is at a certain location on the screen and in a certain orientation. These selective responses to stimuli define each neuron's receptive field properties. (After Purves, Augustine, et al., 2008)

Important as these observations were, amassing this foundational body of information about the response properties of visual neurons was not Hubel and Wiesel's only contribution. At each stage of their investigations, they used imaginative and often new anatomical methods to explore the organization of the thalamus, the primary visual cortex, and some of the higher-order visual processing regions. They also made basic contributions to understanding cortical development as they went along, work that might eventually stand as their greatest legacy. Hubel and Wiesel knew from the studies just described that neurons in V1 are normally innervated by thalamic inputs that can be activated by stimulating the right eye, the left eye, or both eyes (Figure 7.5). What would happen to the neural connections in the cortex if one eye of an experimental animal was closed during early development, depriving the animal of normal visual experience through that eye? Although most of the neurons in V1 are activated to some degree by both eyes (Figure 7.5A), when they closed one eye of a kitten early in life and studied the brain after the animal had matured (which takes about six months in cats), they found a remarkable change. Electrophysiological recordings showed that very few neurons could be driven from the deprived eye: Most of the cortical cells were now being driven by the eye that had remained open (Figure 7.5B). Moreover, the cats were behaviorally blind to stimuli presented to the deprived eye, a deficit that did not resolve even if the deprived eye was subsequently left open for months. The same manipulation in an adult cat—closing one eye for a long period—had no effect on the responses of the visual neurons. Even when they closed one eye for a year or more, the distribution of V1 neurons driven by one eye and the animals' visual behavior tested through the reopened eye were indistinguishable from normal (Figure 7.5C). Therefore, between the time a kitten's eyes open (about a week after birth) and a year of age, visual experience determines how the visual cortex is wired, and does so in a way that later experience does not readily reverse.

Figure 7.5 The effect on cortical neurons of closing one eye in a kitten. A) The distribution observed in the primary visual cortex of normal adult cats by stimulating one eye or the other. Cells in group 1 are activated exclusively by one eye (referred to here as the contralateral eye), and cells in group 7 are activated exclusively by the other (ipsilateral) eye. Neurons in the other groups are activated to varying degrees by both eyes (NR indicates neurons that could not be activated by either eye). B) Following closure of one eye from one week after birth until about two and a half months of age, no cells could be activated by the deprived (contralateral) eye. C) In contrast, a much longer period of monocular deprivation in an adult cat (from 12 to 38 months of age in this example) had little effect on ocular dominance. (After Purves, Augustine, et al., 2008)

The clinical, educational, and social implications of these results are hard to miss. In terms of clinical ophthalmology, early deprivation in developed countries is most often the result of strabismus, a misalignment of the two eyes caused by deficient control of the direction of gaze by the muscles that move the eye. This

problem affects about 5% of children. Because the resulting misalignment produces double vision, the response of the visual system in severely afflicted children is to suppress the input from one eye (it's unclear exactly how this happens). This effect can eventually render children blind in the suppressed eye if they are not treated promptly by intermittently patching the good eye or intervening surgically to realign the eyes. A prevalent cause of visual deprivation in children in underdeveloped countries is a cataract (opacification of the lens) caused by diseases such as river blindness (an infection caused by a parasitic worm) or trachoma (an infection caused by a small, bacteria-like organism). A cataract in one eye is functionally equivalent to monocular deprivation in experimental animals, and this defect also results in an irreversible loss of visual acuity in the untreated child's deprived eye, even if the cataract is later removed. Hubel and Wiesel's observations provided a basis for understanding all this. In keeping with their findings in experimental animals, it was also well known that individuals deprived of vision as adults, such as by accidental corneal scarring, retain the ability to see when treated by corneal transplantation, even if treatment is delayed for decades.

The broader significance of this work for brain function is also readily apparent. If the visual system is a reasonable guide to the development of the rest of the brain, then innate mechanisms establish the initial wiring of neural systems, but normal experience is needed to preserve, augment, and adjust the neural connectivity present at birth. In the case of abnormal experience, such as monocular deprivation, the mechanisms that enable the normal maturation of connectivity are thwarted, resulting in anatomical and, ultimately, behavioral changes that become increasingly hard to reverse as animals grow older. This gradually diminishing cortical plasticity as we or other animals mature provides a neurobiological basis for the familiar observation that we learn anything (language, music, athletic skills, cultural norms) much better as children than as adults, and that behavior is much more susceptible to normal or pathological modification early in development than later. The implications of these further insights for early education, for learning and remediation at later stages of life, and for the legal policies are self-evident.

Hubel and Wiesel's extraordinary success (Figure 7.6) was no doubt the result of several factors. First, as they were always quick to say, they were lucky enough to have come together as fellows in Kuffler's lab shortly after he had determined the receptive field properties of neurons in the cat retina—the approach that, with Kuffler's encouragement, they pursued as Kuffler followed other interests (an act of generosity not often seen when mentors latch on to something important). Second, they were aware of and dedicated to the importance of what they were doing; the experiments were difficult and often ran late into the night, requiring an uncommon work ethic that their medical training helped provide. Finally, they respected and complemented each other as equal partners. Hubel was the more eccentric of the two, and I always found him somewhat daunting. He had been an honors student in math and physics at McGill, and whether solving the Rubik's cube that was always lying around the lunchroom or learning how to program the seemingly incomprehensible PDP 11 computer that he had purchased for the lab, he liked puzzles and logical challenges. He asked tough and highly original questions in seminars or lunchroom conversations and made everyone a little uneasy by taking snapshots with a miniature camera about the size of a cigarette lighter that he carried around. He was hard to talk to when I sought him out for advice as a postdoc, and I couldn't help feeling that his characterization of lesser lights as "chuckleheads" was probably being applied to me. These quirks aside, he is the neuroscientist I have most admired over the years.

Figure 7.6 David Hubel and Torsten Wiesel talking to reporters in 1981, when they were awarded that year's Nobel Prize in Physiology or Medicine. (From Purves and Litchman, 1985)

Although Wiesel shared Hubel's high intelligence and dedication to the work they were doing, he was otherwise quite different. Open and friendly with everyone, he had all the characteristics of the natural leader of any collective enterprise. Torsten became the chair of the Department of Neurobiology at Harvard when Kuffler stepped down in 1973 and, after moving to Rockefeller University in 1983, was eventually appointed president there, a post he served in with great success from 1992 until his retirement in 1998 at the age of 74. In contrast, Hubel had been appointed chair of the Department of Physiology at Harvard in 1967, but he quit after only a few months and returned to the Department of Neurobiology when he apparently discovered that

he did not want to handle all the problems that being a chair entails. (Other reasons might have contributed, based on the response of the Department of Physiology faculty to his managerial style, but if so, I never heard them discussed.)

This brief summary of what Hubel and Wiesel achieved gives some idea of why their influence on the trajectory of "systems-level" neuroscience in the latter decades of the twentieth century was so great. The wealth of evidence they amassed seemed to confirm Helmholtz's idea that perceptions are the result of the activity of neurons that effectively detect and, in some sense, report and represent in the brain the various features of retinal images. This strategy seems eminently logical; any sensible engineer would presumably want to make what we see correspond to the real-world features of the objects that we and other animals must respond to with visually guided behavior. This was the concept of vision that I took away from the course that Hubel and Wiesel taught us postdocs and students in the early 1970s. However, I should hasten to add that feature detection as an explicit goal of visual processing was never discussed. Hubel and Wiesel appeared to assume that understanding the receptive field properties of visual neurons would eventually explain perception, and that further discussion would be superfluous.

In light of all this, it will seem odd that the rest of the book is predicated on the belief that these widely accepted ideas about how the visual brain works are wrong. The further conclusion that understanding what we see based on learning more about the responses of visual neurons is likely to be a dead end might seem even stranger. Several things conspired to sow seeds of doubt after years of enthusiastic, if remote, acceptance of the overall program that Hubel and Wiesel had been pursuing. The first flaw was the increasing difficulty that they and their many acolytes were having when trying to make sense of the electrophysiological and anatomical information that had accumulated by the 1990s. In the early stages of their work, the results obtained seemed to beautifully confirm the intuition that vision entails sequential and essentially hierarchical analyses of retinal image features leading to the neural correlates of perception (see Figure 7.3). The general idea was that the luminance values, spectral distributions (colors), angles, line lengths, depth, motion, and other features were abstracted by visual processing in the retina, thalamus, and primary visual cortex, and subsequently recombined in increasingly complex ways by neurons at progressively higher stages in the visual cortex. These combined representations in the extrastriate regions of the visual system would lead to the perception of objects and their qualities by virtue of further activity elicited in the association cortices in the occipital lobes and adjacent areas in the temporal and parietal lobes.

A particularly impressive aspect of Hubel and Wiesel's observations in the 1960s and 1970s was that the receptive field properties of the neurons in the lateral geniculate nucleus of the thalamus could nicely explain the properties of the neurons they contacted in the input layer of the primary visual cortex, and that the properties of these neurons could explain the responses of the neurons they contacted at the next higher level of processing in V1. The neurons in this cortical hierarchy were referred to as "simple," "complex," and "hypercomplex" cells, underscoring the idea that the features abstracted from the retinal image were progressively being put back together in the cortex for the purpose of perception. Although I doubt Hubel and Wiesel ever used the phrase, the rationale for the initial abstraction was generally assumed to be engineering or coding efficiency.

These findings also fit well with their anatomical evidence that V1 is divided into iterated modules defined by particular response properties, such as selectivity for orientation (see Figure 7.4) or for information related to the left or right eye (see Figure 6.2A). By the late 1970s, Hubel and Wiesel had put these several findings together in what they called the "ice cube" model of visual cortical processing (Figure 7.7). The suggestion was that each small piece of cortex, which they called a "hyercolumn," contained a complete set of feature-processing elements. But as the years passed and more evidence accumulated about visual neuronal types, their connectivity, and the organization of the visual system, the concept of a processing hierarchy in general and the ice cube model in particular seemed as if a square peg was being pounded into a round hole.

Figure 7.7 The ice cube model of primary visual cortical organization. This diagram illustrates the idea that units roughly a square millimeter or two in size (the primary visual cortex in each hemisphere of a rhesus monkey brain is about 1,000 square millimeters) each comprise superimposed feature-processing elements,

illustrated here by orientation selectivity over the full range of possible angles (the little lines) and comapping with right and left eye processing stripes (indicated by L and R; see Figure 6.2A). (After Hubbel, 1988)

A second reason for suspecting that more data about the receptive field properties of visual neurons and their anatomical organization might not explain perception was the mountain of puzzling observations about what people actually see, coupled with philosophical concerns about vision that had been around for centuries. Taking such things seriously was a path that a self-respecting neuroscientist followed at some peril. But vision has always demanded that perceptual and philosophical issues be considered, and the cracks that had begun to appear in the standard model of how the visual brain was supposed to work encouraged a reconsideration of some basic concerns. One widely discussed issue was the question of "grandmother cells," a term coined by Jerry Lettvin, an imaginative and controversial neuroscientist at MIT who liked the role of intellectual and (during the Vietnam War era) social provocateur. If the features of retinal images were being progressively put back together in neurons with increasingly more complex properties at higher levels of the brain, didn't this imply the existence of nerve cells that would ultimately be ludicrously selective (meaning neurons that would respond to only the retinal image of your grandmother, for example)? Although the question was facetious, many people correctly saw it as serious. The ensuing debate was further stimulated by the discovery in the early 1980s of neurons in the association areas of the monkey brain that did, in fact, respond specifically to faces (an area in the human temporal lobe that responds selectively to faces has since been well documented). A related question concerned the binding problem. Even if visual neurons don't generate perceptions by specifically responding to grandmothers or other particular objects (which most people agreed made little sense), how are the various features of any object brought together in a coherent, instantaneously generated perception of, for example, a ball that is round, chartreuse, and coming at you in a particular direction from a certain distance at a certain speed (think tennis). Although purported answers to the binding problem were (and still are) taken with a grain of salt, most neuroscientists recognized that such questions would eventually need to be answered. Although a lot of my colleagues were not very interested in debates of this sort, I had always had a weakness for them and was glad to see these issues raised as serious concerns in neuroscience. After all, I had been a philosophy major in college and had left clinical medicine because I wanted to understand how the brain worked, not just how to understand its maladies or the properties of its constituent cells.

By the mid-1990s, I began to be bothered by another philosophical issue relevant to perception that was ultimately decisive in reaching the conclusion that mining the details of visual neuronal properties would never lead to an understanding of perception or its underlying mechanics. Western philosophy had long debated about how the "real world" of physical objects can be "known" by using our senses. Positions on this issue had varied greatly, the philosophical tension in recent centuries being between thinkers such as Francis Bacon and René Descartes, who supposed that absolute knowledge of the real world is possible (an issue of some scientific consequence in modern physics and cosmology), and others such as David Hume and Immanuel Kant, who argued that the real world is inevitably remote from us and can be appreciated only indirectly. The philosopher who made these points most cogently with respect to vision was George Berkeley, an Irish nobleman, bishop, tutor at Trinity College in Dublin, and card-carrying member of the British "Empiricist School." In 1709, Berkeley had written a short treatise entitled An Essay Toward a New Theory of Vision in which he pointed out that a two-dimensional image projected onto the receptive surface of the eye could never specify the three-dimensional source of that image in the world (Figure 7.8). This fact and the difficulty it raises for understanding the perception of any image feature is referred to as the inverse optics problem.

Figure 7.8 The inverse optics problem. George Berkeley pointed out in the eighteenth century that the same projected image could be generated by objects of different sizes, at different distances from the observer, and in different physical orientations. As a result, the actual source of any three-dimensional object is inevitably uncertain. Note that the problem is not simply that retinal images are ambiguous; the deeper issue is that the real world is directly unknowable by means of any logical operation on a projected image. (After Purves and Lotto, 2003)

In the context of biology and evolution, the significance of the inverse problem is clear: If the information on the retina precludes direct knowledge of the real world, how is it that what we see enables us to respond so successfully to real-world objects on the basis of vision? Helmholtz was aware of the problem and argued that vision had to depend on learning from experience in addition to the information supplied by neural connections

in the brain determined by inheritance. However, he thought that analyzing image features was generally good enough and that a boost from empirical experience (empirical experience, for him, was what we learn about objects in life through trial-and-error interactions) would contend with the inverse problem. This learned information would allow us to make what Helmholtz referred to as "unconscious inferences" about what an ambiguous image might represent. Some vision scientists seemed to take Helmholtz's approach to the inverse optics problem as sufficient, but many simply ignored it. The problem was rarely, if ever, mentioned in the discussions of vision I had been party to over the years. In particular, I had never heard Hubel and Wiesel mention it or saw it referred to in their papers.

At the same time, I was increasingly aware in the 1990s, as anyone who delves into perception must be, of an enormous number of visual illusions. An illusion refers to a perception that fails to match a physical measurement made by using an instrument of some sort: a ruler, a protractor, a photometer, or some more complex device that makes direct measurements of object properties, therefore evading the inverse problem. In using the term illusion the presumption in psychology texts and other literature is that we usually see the world "correctly," but sometimes a natural or contrived stimulus fools us so that our perception and the measured reality underlying the stimulus fail to align. But if what Berkeley had said was right, analysis of a retinal image could not tell the brain anything definite about what objects and conditions in the world had actually generated an image. It seemed more likely that all perceptions were equally illusory constructions produced by the brain to achieve biological success in the face of the inverse problem. If this was the case, then the evolution of visual systems must have been primarily concerned with solving this fundamental challenge. Surprisingly, no one seemed to be paying much attention to this very large spanner that Berkeley had tossed into logical and analytical concepts of how vision works.

I didn't have the slightest idea of how the visual wiring described by Hubel and Wiesel and their followers might be contending with the inverse problem. But I was pretty sure that it must be by means of a very different strategy from the one that had been explicitly or implicitly dominating my thinking (and most everyone else's) since the 1960s. If understanding brain function was going to be possible, exploring how vision contends with the inverse problem seemed a very good place to start.

w w w . s c i a m . c o m S C I E N T I F I C A M E R I C A N 53

DAV

ID E

MM

ITE

Jones and Coleman are among a handful of otherwise normalpeople who have synesthesia. They experience the ordinaryworld in extraordinary ways and seem to inhabit a mysteriousno-man’s-land between fantasy and reality. For them the sens-es—touch, taste, hearing, vision and smell—get mixed up in-stead of remaining separate.

Modern scientists have known about synesthesia since1880, when Francis Galton, a cousin of Charles Darwin, pub-lished a paper in Nature on the phenomenon. But most havebrushed it aside as fakery, an artifact of drug use (LSD andmescaline can produce similar effects) or a mere curiosity.About four years ago, however, we and others began to un-cover brain processes that could account for synesthesia. Alongthe way, we also found new clues to some of the most mysteri-ous aspects of the human mind, such as the emergence of ab-stract thought, metaphor and perhaps even language.

A common explanation of synesthesia is that the affectedpeople are simply experiencing childhood memories and asso-ciations. Maybe a person had played with refrigerator magnets

as a child and the number 5 was red and 6 was green. This the-ory does not answer why only some people retain such vividsensory memories, however. You might think of cold when youlook at a picture of an ice cube, but you probably do not feelcold, no matter how many encounters you may have had withice and snow during your youth.

Another prevalent idea is that synesthetes are merely beingmetaphorical when they describe the note C flat as “red” or saythat chicken tastes “pointy”—just as you and I might speak ofa “loud” shirt or “sharp” cheddar cheese. Our ordinary lan-guage is replete with such sense-related metaphors, and perhapssynesthetes are just especially gifted in this regard.

We began trying to find out whether synesthesia is a gen-uine sensory experience in 1999. This deceptively simple ques-tion had plagued researchers in this field for decades. One nat-ural approach is to start by asking the subjects outright: “Is thisjust a memory, or do you actually see the color as if it were rightin front of you?” When we tried asking this question, we didnot get very far. Some subjects did respond, “Oh, I see it per-

People with synesthesia—whose sensesblend together—are providing valuable clues

to understanding the organization andfunctions of the human brain

By Vilayanur S. Ramachandran and Edward M. Hubbard

When Matthew Blakeslee shapes hamburger patties with his hands, he experiencesa vivid bitter taste in his mouth. Esmerelda Jones (a pseudonym) sees blue whenshe listens to the note C sharp played on the piano; other notes evoke differenthues—so much so that the piano keys are actually color-coded, making it easierfor her to remember and play musical scales. And when Jeff Coleman looks atprinted black numbers, he sees them in color, each a different hue. Blakeslee,

Hearing Colors, Tasting Shapes

fectly clearly.” But a more frequent reac-tion was, “I kind of see it, kind of don’t”or “No, it is not like a memory. I see thenumber as being clearly red but I alsoknow it isn’t; it’s black. So it must be amemory, I guess.”

To determine whether an effect is tru-ly perceptual, psychologists often use asimple test called pop-out or segregation.If you look at a set of tilted lines scatteredamid a forest of vertical lines, the tiltedlines stand out. Indeed, you can instantlysegregate them from the background andgroup them mentally to form, for exam-ple, a separate triangular shape. Similar-ly, if most of a background’s elementswere green dots and you were told to lookfor red targets, the reds would pop out.On the other hand, a set of black 2’s scat-tered among 5’s of the same color almostblend in [see illustration on page 57]. It ishard to discern the 2’s without engagingin an item-by-item inspection of numbers,even though any individual number is justas clearly different from its neighbors as atilted line is from a straight line. We thusmay conclude that only certain primitive,or elementary, features, such as color andline orientation, can provide a basis forgrouping. More complex perceptual to-kens, such as numbers, cannot do so.

We wondered what would happen ifwe showed the mixed numbers to synes-thetes who experience, for instance, redwhen they see a 5 and green with a 2. Wearranged the 2’s so that they formed a tri-angle. If synesthesia were a genuine sen-sory effect, our subjects should easily seethe triangle because for them, the num-bers would look colored.

When we conducted pop-out tests

with volunteers, the answer was crystalclear. Unlike normal subjects, synesthetescorrectly reported the shape formed bygroups of numbers up to 90 percent of thetime (exactly as nonsynesthetes do whenthe numbers actually have different col-ors). This result proves that the inducedcolors are genuinely sensory and thatsynesthetes are not just making things up.It is impossible for them to fake their suc-cess. In another striking example, weasked a synesthete who sees 5 tinged redto watch a computer display. He couldnot tell when we surreptitiously added anactual red hue to the white number unlessthe red was sufficiently intense; he couldinstantly spot a real green added to the 5.

Visual ProcessingCONFIRMATION THAT synesthesia isreal brings up the question, Why do somepeople experience this weird phenome-non? Our experiments lead us to favorthe idea that synesthetes are experiencingthe result of some kind of cross wiring inthe brain. This basic concept was initial-ly proposed about 100 years ago, but wehave now identified where in the brainand how such cross wiring might occur.

An understanding of the neurobio-logical factors at work requires some fa-miliarity with how the brain processes vi-sual information [see illustration on op-posite page]. After light reflected from ascene hits the cones (color receptors) inthe eye, neural signals from the retinatravel to area 17, in the occipital lobe atthe back of the brain. There the image isprocessed further within local clusters, orblobs, into such simple attributes as col-or, motion, form and depth. Afterward,

information about these separate fea-tures is sent forward and distributed toseveral far-flung regions in the temporaland parietal lobes. In the case of color,the information goes to area V4 in thefusiform gyrus of the temporal lobe.From there it travels to areas that lie far-ther up in the hierarchy of color centers,including a region near a patch of cortexcalled the TPO (for the junction of thetemporal, parietal and occipital lobes).These higher areas may be concernedwith more sophisticated aspects of colorprocessing. For example, leaves look asgreen at dusk as they do at midday, eventhough the mix of wavelengths reflectedfrom the leaves is very different.

Numerical computation, too, seems tohappen in stages. An early step also takesplace in the fusiform gyrus, where the ac-tual shapes of numbers are represented,and a later one occurs in the angular gyrus,a part of the TPO that is concerned withnumerical concepts such as ordinality (se-quence) and cardinality (quantity). (Whenthe angular gyrus is damaged by a strokeor a tumor, the patient can still identifynumbers but can no longer divide or sub-tract. Multiplication often survives be-cause it is learned by rote.) In addition,brain-imaging studies in humans strong-ly hint that visually presented letters ofthe alphabet or numbers (graphemes) ac-tivate cells in the fusiform gyrus, where-as the sounds of the syllables (phonemes)are processed higher up, once again in thegeneral vicinity of the TPO.

Because both colors and numbers areprocessed initially in the fusiform gyrusand subsequently near the angular gyrus,we suspected that number-color synesthe-sia might be caused by cross wiring be-tween V4 and the number-appearancearea (both within the fusiform) or be-tween the higher color area and the num-ber-concept area (both in the TPO). Oth-er, more exotic forms of the conditionmight result from similar cross wiring ofdifferent sensory-processing regions. Thatthe hearing center in the temporal lobesis also close to the higher brain area thatreceives color signals from V4 could ex-plain sound-color synesthesia. Similarly,Matthew Blakeslee’s tasting of touchmight occur because of cross wiring be-

54 S C I E N T I F I C A M E R I C A N M A Y 2 0 0 3

■ Synesthesia (from the Greek roots syn, meaning “together,” and aisthesis, or“perception”) is a condition in which otherwise normal people experience theblending of two or more senses.

■ For decades, the phenomenon was often written off as fakery or simplymemories, but it has recently been shown to be real. Perhaps it occurs becauseof cross activation, in which two normally separate areas of the brain elicitactivity in each other.

■ As scientists explore the mechanisms involved in synesthesia, they are alsolearning about how the brain in general processes sensory information anduses it to make abstract connections between seemingly unrelated inputs.

Overview/Synesthesia

tween the taste cortex in a region calledthe insula and an adjacent cortex repre-senting touch by the hands.

Assuming that neural cross wiringdoes lie at the root of synesthesia, whydoes it happen? We know that it runs infamilies, so it has a genetic component.Perhaps a mutation causes connections toemerge between brain areas that are usu-ally segregated. Or maybe the mutationleads to defective pruning of preexistingconnections between areas that are nor-mally connected only sparsely. If the mu-tation were to be expressed (that is, to ex-ert its effects) in some brain areas but notothers, this patchiness might explain whysome synesthetes conflate colors and num-

bers whereas others see colors when theyhear phonemes or musical notes. Peoplewho have one type of synesthesia are morelikely to have another, which adds weightto this idea.

Although we initially thought in termsof physical cross wiring, we have come torealize that the same effect could occur ifthe wiring—the number of connectionsbetween regions—was fine but the balanceof chemicals traveling between regionswas skewed. So we now speak in terms ofcross activation. For instance, neighboringbrain regions often inhibit one another’sactivity, which serves to minimize crosstalk. A chemical imbalance of some kindthat reduces such inhibition—for example,

by blocking the action of an inhibitoryneurotransmitter or failing to produce aninhibitor—would also cause activity inone area to elicit activity in a neighbor.Such cross activation could, in theory, alsooccur between widely separated areas,which would account for some of the lesscommon forms of synesthesia.

Support for cross activation comesfrom other experiments, some of whichalso help to explain the varied formssynesthesia can take. One takes advan-tage of a visual phenomenon known ascrowding [see illustration on oppositepage]. If you stare at a small plus sign inan image that also has a number 5 off toone side, you will find that it is easy to dis-

w w w . s c i a m . c o m S C I E N T I F I C A M E R I C A N 55

CAR

OL

DO

NN

ER

(il

lust

rati

on);

GE

OFF

BO

YNTO

N S

alk

In

stit

ute

for

Bio

log

ica

l St

ud

ies

AND

VIL

AYAN

UR

S.

RAM

ACH

AND

RAN

AN

D E

DW

ARD

M.

HU

BB

ARD

(in

set)

MINGLED SIGNALSIN ONE OF THE MOST COMMON FORMS of synesthesia, looking at a number evokes a specifichue. This apparently occurs because brain areas that normally do not interact whenprocessing numbers or colors do activate each other in synesthetes.

REAR VIEW of a synesthete’s brain, made with functional magnetic resonanceimaging, shows high activity (yellow) in the V4 color-processing area as thesubject looks at white numbers on a gray background. This area is not activein people with normal perception viewing the same figures.

NEURAL SIGNALS from the retinatravel via optic radiation to area17, in the rear of the brain, wherethey are broken into simpleshared attributes such as color,form, motion and depth.

Color information continueson to V4, near where the visualappearance of numbers is alsorepresented—and thus is a site forcross-linking between the colorand number areas (short pink andgreen arrows).

Ultimately, color proceeds“higher,” to an area near the TPO(for temporal, parietal, occipitallobes) junction, which mayperform more sophisticated colorprocessing. Similarly, a laterstage of numerical computationoccurs in the angular gyrus, apart of the TPO concerned withthe concepts of sequence andquantity. This could explainsynesthesia in people who linkcolors with abstract numericalsequences, like days of the week.

TPO JUNCTION

OCCIPITAL LOBE

AREA 17

V4

NUMBER- APPEARANCE AREATEMPORAL LOBE

OPTIC NERVE

V4

LIGHT

RETINA

PARIETAL LOBE

OPTIC RADIATION

cern that number, even though you arenot looking at it directly. But if we nowsurround the 5 with four other numbers,such as 3’s, then you can no longer iden-tify it. It looks out of focus. Volunteerswho perceive normally are no more suc-cessful at identifying this number thanmere chance. That is not because thingsget fuzzy in the periphery of vision. Afterall, you could see the 5 perfectly clearlywhen it wasn’t surrounded by 3’s. Youcannot identify it now because of limitedattentional resources. The flanking 3’ssomehow distract your attention awayfrom the central 5 and prevent you fromseeing it.

A big surprise came when we gave thesame test to two synesthetes. They looked

at the display and made remarks like, “Icannot see the middle number. It’s fuzzybut it looks red, so I guess it must be a 5.”Even though the middle number did notconsciously register, it seems that the brainwas nonetheless processing it somewhere.Synesthetes could then use this color to de-duce intellectually what the number was.If our theory is right, this finding impliesthat the number is processed in thefusiform gyrus and evokes the appropri-ate color before the stage at which thecrowding effect occurs in the brain; para-doxically, the result is that even an “in-visible” number can produce synesthesia.

Another finding we made also sup-ports this conclusion. When we reducedthe contrast between the number and the

background, the synesthetic color be-came weaker until, at low contrast, sub-jects saw no color at all, even though thenumber was perfectly visible. Whereasthe crowding experiment shows that aninvisible number can elicit color, the con-trast experiment conversely indicates thatviewing a number does not guaranteeseeing a color. Perhaps low-contrast num-bers activate cells in the fusiform ade-quately for conscious perception of thenumber but not enough to cross-activatethe color cells in V4.

Finally, we found that if we showedsynesthetes Roman numerals, a V, say,they saw no color—which suggests that itis not the numerical concept of a number,in this case 5, but the grapheme’s visualappearance that drives the color. This ob-servation, too, implicates cross activationwithin the fusiform gyrus itself in num-ber-color synesthesia, because that struc-ture is mainly involved in analyzing the vi-sual shape, not the high-level meaning ofthe number. One intriguing twist: Imag-ine an image with a large 5 made up of lit-tle 3’s; you can see either the “forest” (the5) or focus minutely on the “trees” (the3’s). Two synesthete subjects reportedthat they saw the color switch, dependingon their focus. This test implies that even

56 S C I E N T I F I C A M E R I C A N M A Y 2 0 0 3

DAV

ID E

MM

ITE

VILAYANUR S. RAMACHANDRAN and EDWARD M. HUBBARD collaborate on studies of synes-thesia. Ramachandran directs the Center for Brain and Cognition at the University of Califor-nia at San Diego and is adjunct professor at the Salk Institute for Biological Studies. He trainedas a physician and later obtained a Ph.D. from Trinity College, University of Cambridge. He hasreceived a fellowship from All Souls College, University of Oxford, the Ariens Kappers Gold Medalfrom the Royal Netherlands Academy, and the plenary lecture award from the American Acad-emy of Neurology. He gave the BBC Reith Lectures for 2003. This is his fourth article for Sci-entific American. Hubbard is a fourth-year graduate student in the departments of psycholo-gy and cognitive science at U.C.S.D. His research combines psychophysics and functional mag-netic resonance imaging to explore the neural basis of multisensory phenomena. A foundingmember of the American Synesthesia Association, he helped to organize its second annualmeeting at U.C.S.D. in 2001.

THE

AU

THO

RS

though synesthesia can arise as a result ofthe visual appearance alone—not thehigh-level concept—the manner in whichthe visual input is categorized, based onattention, is also critical.

But as we began to recruit other vol-unteers, it soon became obvious that notall synesthetes who colorize their worldare alike. In some, even days of the weekor months of the year elicit colors. Mon-day might be green, Wednesday pink,and December yellow.

The only thing that days of the week,months and numbers have in common isthe concept of numerical sequence, or or-dinality. For certain synesthetes, perhapsit is the abstract concept of numerical se-quence that drives the color, rather thanthe visual appearance of the number.Could it be that in these individuals, thecross wiring occurs between the angulargyrus and the higher color area near theTPO instead of between areas in thefusiform? If so, that interaction wouldexplain why even abstract number rep-resentations, or the idea of the numberselicited by days of the week or months,will strongly evoke specific colors. In oth-er words, depending on where in thebrain the mutant gene is expressed, it canresult in different types of the condition—

“higher” synesthesia, driven by numericalconcept, or “lower” synesthesia, pro-duced by visual appearance alone. Simi-larly, in some lower forms, the visual ap-pearance of a letter might generate color,whereas in higher forms it is the sound, orphoneme, summoned by that letter; pho-nemes are represented near the TPO.

We also observed one case in whichwe believe cross activation enables a color-blind synesthete to see numbers tingedwith hues he otherwise cannot perceive;charmingly, he refers to these as “Mar-tian colors.” Although his retinal colorreceptors cannot process certain wave-lengths, we suggest that his brain colorarea is working just fine and being cross-activated when he sees numbers.

In brain-imaging experiments we areconducting with Geoff Boynton of theSalk Institute for Biological Studies in SanDiego, we have obtained preliminary ev-idence of local activation of the color areaV4 in a manner predicted by our cross-

activation theory of synesthesia. (JeffreyGray of the Institute of Psychiatry in Lon-don and his colleagues have reported sim-ilar results.) On presenting black andwhite numbers to synesthetes, brain acti-vation arose not only in the numberarea—as it would in normal subjects—butalso in the color area. Our group also ob-served differences between types of synes-thetes. One of our subjects with lowersynesthesia showed much greater activa-tion in earlier stages of color processingthan occurred in controls. In contrast,higher synesthetes show less activation atthese earlier levels.

A Way with MetaphorO U R I N S I G H T S into the neurologicalbasis of synesthesia could help explainsome of the creativity of painters, poets

and novelists. According to one study, thecondition is seven times as common in cre-ative people as in the general population.

One skill that many creative peopleshare is a facility for using metaphor (“Itis the east, and Juliet is the sun”). It is asif their brains are set up to make links be-tween seemingly unrelated domains—

such as the sun and a beautiful youngwoman. In other words, just as synesthe-sia involves making arbitrary links be-tween seemingly unrelated perceptual en-tities such as colors and numbers, meta-phor involves making links betweenseemingly unrelated conceptual realms.Perhaps this is not just a coincidence.

Numerous high-level concepts areprobably anchored in specific brain re-gions, or maps. If you think about it, thereis nothing more abstract than a number,

w w w . s c i a m . c o m S C I E N T I F I C A M E R I C A N 57

VILA

YAN

UR

S.

RAM

ACH

AND

RAN

COLOR-CODED WORLDIN A TEST of visual-segregation capabilities, synesthetes who link a specific huewith a given number can instantly see an embedded pattern in an image with blacknumbers scattered on a white page. Whereas a person with normal perceptionmust undertake a digit-by-digit search to pick out, in this example, 2’s amid 5’s(left), the triangle-shaped group of 2’s pops out for a synesthete (right).

“INVISIBLE” NUMBERS show up for synesthetes in a perceptual test. When a personstares at a central object, here a plus sign, a single digit off to one side is easy tosee with peripheral vision (left). But if the number is surrounded by others (right),it appears blurry—invisible—to the average person. In contrast, a synesthete coulddeduce the central number by the color it evokes.

and yet it is represented, as we have seen,in a relatively small brain region, the an-gular gyrus. Let us say that the mutationwe believe brings about synesthesia caus-es excess communication among differ-ent brain maps—small patches of cortexthat represent specific perceptual entities,such as sharpness or curviness of shapesor, in the case of color maps, hues. De-pending on where and how widely in thebrain the trait was expressed, it couldlead to both synesthesia and to a propen-sity toward linking seemingly unrelatedconcepts and ideas—in short, creativity.This would explain why the apparentlyuseless synesthesia gene has survived inthe population.

In addition to clarifying why artistsmight be prone to experiencing synesthe-sia, our research suggests that we all havesome capacity for it and that this traitmay have set the stage for the evolution ofabstraction—an ability at which humansexcel. The TPO (and the angular gyruswithin it), which plays a part in the con-dition, is normally involved in cross-modal synthesis. It is the brain regionwhere information from touch, hearingand vision is thought to flow together toenable the construction of high-level per-ceptions. For example, a cat is fluffy(touch), it meows and purrs (hearing), ithas a certain appearance (vision) andodor (smell), all of which are derived si-multaneously by the memory of a cat orthe sound of the word “cat.”

Could it be that the angular gyrus—

which is disproportionately larger in hu-mans compared with that in apes andmonkeys—evolved originally for cross-modal associations but then became co-opted for other, more abstract functionssuch as metaphors? Consider two draw-ings, originally designed by psychologistWolfgang Köhler. One looks like an ink-blot and the other, a jagged piece of shat-tered glass. When we ask, “Which of theseis a ‘bouba,’ and which is a ‘kiki’?” 98 per-cent of people pick the inkblot as a boubaand the other one as a kiki. Perhaps that isbecause the gentle curves of the amoeba-like figure metaphorically mimic the gen-tle undulations of the sound “bouba” asrepresented in the hearing centers in thebrain as well as the gradual inflection of

the lips as they produce the curved “boo-baa” sound. In contrast, the waveform ofthe sound “kiki” and the sharp inflectionof the tongue on the palate mimic thesudden changes in the jagged visualshape. The only thing these two kiki fea-tures have in common is the abstractproperty of jaggedness that is extractedsomewhere in the vicinity of the TPO,probably in the angular gyrus. (We re-cently found that people with damage tothe angular gyrus lose the bouba-kiki ef-fect—they cannot match the shape withthe correct sound.) In a sense, perhaps weare all closet synesthetes.

So the angular gyrus performs a veryelementary type of abstraction—extract-ing the common denominator from a setof strikingly dissimilar entities. We do

IMAGINE A BAND of ancestral hominids about toinvent language. Clearly, they did not begin byhaving a leader say, “Hey, look at this—let’s callit a banana. All of you say after me, ba-na-na.”Undoubtedly, though, the group had a set ofcapacities that prepared the ground forsystematic verbal communication. Our studiesof the neurobiological basis of synesthesiasuggest that a facility for metaphor—for seeingdeep links between superficially dissimilar andunrelated things—provided a key seed for theeventual emergence of language.

Humans have a built-in bias to associatecertain sounds with particular visual shapes,which could well have been important in gettinghominids started on a shared vocabulary. Inaddition, specific brain areas that process visualshapes of objects, letters and numbers, andword sounds can activate each other even innonsynesthetes, causing people to expect, say,jagged shapes to have harsh-sounding names.

Two other types of neural connectionssupport our idea. First, the sensory areas forvisual shapes and for hearing in the back of thebrain can cross-activate specific motor areas inthe front of the brain that participate in speech.A sharp visual inflection or a harsh soundinduces the motor control area for speech to

Synesthesia may provide someinsights about the evolution of thought and language

COMMON QUESTIONS Are there different types of synesthesia?Science counts more than 100. The conditionruns in families and may be more common inwomen and creative people; perhaps oneperson in 200 has synesthesia. In the mostprevalent type, looking at numbers or listeningto tones evokes colors. In one rare kind, eachletter is associated with the male or femalesex—an example of the brain’s tendency tosplit the world into binary categories.

If a synesthete associates a color with asingle letter or number, what happens if helooks at a pair of letters, such as “ea,” ordouble digits, as in “25”?He sees colors that correspond with theindividual letters and numbers. If the letters ornumbers are too close physically, however,they may cancel each other out (colordisappears) or, if the two happen to elicit thesame color, enhance each other.

Does it matter whether letters areuppercase or lowercase?In general, no. But people have sometimesdescribed seeing less saturated color inlowercase letters, or the lowercase lettersmay appear shiny or even patchy.

How do entire words look?Often the color of the first letter spreadsacross the word; even silent letters, such asthe “p” in “psalm,” cause this effect.

What if the synesthete is multilingual?One language can have colored graphemes,but a second (or additional others) may not,perhaps because separate tongues arerepresented in different brain regions.

What about when the person mentallypictures a letter or number?Imagining can evoke a stronger color thanlooking at a real one. Perhaps that exerciseactivates the same brain areas as doesviewing real colors—but because nocompeting signals from a real number arecoming from the retina, the imagined onecreates a stronger synesthetic color.

Does synesthesia improve memory?It can. The late Russian neurologistAleksandr R. Luria described a mnemonistwho had remarkable recall because all of hisfive senses were linked. Even having twolinked senses may help. —V.S.R. and E.M.H.

M A Y 2 0 0 3

not know how exactly it does this job.But once the ability to engage in cross-modal abstraction emerged, it mighthave paved the way for the more com-plex types of abstraction. The oppor-tunistic takeover of one function for adifferent one is common in evolution.For example, bones in the ear used forhearing in mammals evolved from theback of the jawbone in reptiles. Beyondmetaphor and abstract thinking, cross-modal abstraction might even have pro-vided seeds for language [see box above].

When we began our research onsynesthesia, we had no inkling of whereit would take us. Little did we suspectthat this eerie phenomenon, long regard-ed as a mere curiosity, might offer a win-dow into the nature of thought.

w w w . s c i a m . c o m S C I E N T I F I C A M E R I C A N 59

VILA

YAN

UR

S.

RAM

ACH

AND

RAN

The Man Who Tasted Shapes. R. E. Cytowic. MIT Press, 1993. Synaesthesia: Classic and Contemporary Readings. S. Baron-Cohen and J. E. Harrison. Blackwell, 1997.Psychophysical Investigations into the Neural Basis of Synaesthesia. V. S. Ramachandran and E. M. Hubbard in Proceedings of the Royal Society of London, B, Vol. 268, pages 979–983; 2001.Synaesthesia: A Window into Perception, Thought and Language. V. S. Ramachandran and E. M. Hubbard in Journal of Consciousness Studies, Vol. 8, No. 12, pages 3–34; 2001.Synaesthetic Photisms Influence Visual Perception. D. Smilek, M. J. Dixon, C. Cudahy and M. Meriklein Journal of Cognitive Neuroscience, Vol. 13, No. 7, pages 930–936; 2001. Functional Magnetic Resonance Imaging of Synesthesia: Activation of V4/V8 by Spoken Words.J. A. Nunn, L. J. Gregory, M. Brammer, S.C.R. Williams, D. M. Parslow, M. J. Morgan, R. G. Morris, E. T. Bullmore, S. Baron-Cohen and J. A. Gray in Nature Neuroscience, Vol. 5, pages 371–375; 2002.For more on synesthetia, visit www.sciam.com/ontheweb

M O R E T O E X P L O R E

produce an equally sudden inflection of thetongue on the palate (or consider the spokenwords “diminutive,” “teeny-weeny” and “unpeu,” which involve pursing the lips to mimicthe small size of the object. The brain seemsto possess preexisting rules for translatingwhat we see and hear into mouth motionsthat reflect those inputs.

Second, a kind of spillover of signalsoccurs between two nearby motor areas:those that control the sequence of musclemovements required for hand gestures andthose for the mouth. We call this effect“synkinesia.” As Charles Darwin pointed out,when we cut paper with scissors, our jawsmay clench and unclench unconsciously as ifto echo the hand movements. Many linguistsdo not like the theory that manual gesturingcould have set the stage for vocal language,but we believe that synkinesia suggests thatthey may be wrong.

Assume that our ancestral hominidscommunicated mainly through emotionalgrunts, groans, howls and shrieks, which areknown to be produced by the right hemisphereand an area in the frontal lobes concernedwith emotion. Later the hominids developed arudimentary gestural system that becamegradually more elaborate and sophisticated;it is easy to imagine how the hand movementfor pulling someone toward you might haveprogressed to a “come hither” wave. If such

gestures were translated through synkinesiainto movements of the mouth and facemuscles, and if emotional gutturalutterances were channeled through thesemouth and tongue movements, the resultcould have been the first spoken words.

How would we import syntax, the rulesfor using words and phrases in language, intothis scheme? We believe that the evolutionof tool use by hominids may have played animportant role. For example, the tool-building sequence—first shape thehammer’s head, then attach it to a handle,then chop the meat—resembles theembedding of clauses within largersentences. Following the lead of

psychologist Patricia Greenfield of theUniversity of California at Los Angeles, wepropose that frontal brain areas that evolvedfor subassembly in tool use may later havebeen co-opted for a completely novelfunction—joining words into phrases andsentences.

Not every subtle feature of modernlanguage is explained by such schemes, but wesuspect that these elements were critical forsetting in motion the events that culminated in modern language. —V.S.R. and E.M.H.

THE PUZZLE OF LANGUAGEIF ASKED which of the two figures below is a “bouba” and which is a “kiki,” 98percent of all respondents choose the blob as a bouba and the other as a kiki. Theauthors argue that the brain’s ability to pick out an abstract feature in common—such as a jagged visual shape and a harsh-sounding name—could have pavedthe way for the development of metaphor and perhaps even a shared vocabulary.

A broadcast version of this article will air

April 24 on National Geographic Today,

a program on the National Geographic

Channel. Please check your local listings.

Language regions of brain are operativein color perceptionWai Ting Sioka,b, Paul Kayc,d,1, William S. Y. Wange, Alice H. D. Chana,b, Lin Chenf, Kang-Kwong Lukea,b,and Li Hai Tana,b,1

aDepartment of Linguistics and bState Key Laboratory of Brain and Cognitive Sciences, University of Hong Kong, Pokfulam Road, Hong Kong, China;cDepartment of Linguistics, University of California, Berkeley, CA 94720; dInternational Computer Science Institute, 1947 Center Street, Berkeley, CA 94704;eLanguage Engineering Laboratory, Department of Electronic Engineering, Chinese University of Hong Kong, Shatin, Hong Kong, China; and fState KeyLaboratory of Brain and Cognitive Science, Institute of Biophysics, Chinese Academy of Sciences, Beijing 100101, China

Contributed by Paul Kay, April 2, 2009 (sent for review February 11, 2009)

The effect of language on the categorical perception of color isstronger for stimuli in the right visual field (RVF) than in the leftvisual field, but the neural correlates of the behavioral RVF advan-tage are unknown. Here we present brain activation maps reveal-ing how language is differentially engaged in the discrimination ofcolored stimuli presented in either visual hemifield. In a rapid,event-related functional MRI study, we measured subjects’ brainactivity while they performed a visual search task. Compared withcolors from the same lexical category, discrimination of colors fromdifferent linguistic categories provoked stronger and faster re-sponses in the left hemisphere language regions, particularly whenthe colors were presented in the RVF. In addition, activation ofvisual areas 2/3, responsible for color perception, was much stron-ger for RVF stimuli from different linguistic categories than forstimuli from the same linguistic category. Notably, the enhancedactivity of visual areas 2/3 coincided with the enhanced activity ofthe left posterior temporoparietal language region, suggestingthat this language region may serve as a top-down control sourcethat modulates the activation of the visual cortex. These findingsshed light on the brain mechanisms that underlie the hemifield-dependent effect of language on visual perception.

functional magnetic resonance imaging (fMRI) � lateralization

A typically viewed scene permits multiple visual parses, someof which can be readily mapped onto linguistic terms,

whereas others cannot. Does linguistic information play a role invisual perception? For more than half a century, this questionhas provoked controversy. According to the hypothesis proposedby Benjamin Lee Whorf (1), by filtering perception, languageaffects our apprehension of the world. This hypothesis hasreceived conflicting evidence (2–21); a recent review favors theview that linguistic categories filter some, but not all, perceptualinputs and that perceptual factors influence, but do not exclu-sively determine, linguistic categories of color (22).

Recent neuropsychological investigations examining visualfield asymmetries in the categorical perception (CP) of colorshave provided a new perspective on Whorfian effects. In a studyusing a visual search task (7), adult English speakers wererequired to detect a single target color among 11 identicaldistractor colors. Response times for finding the target werefaster when target and distractors were from 2 different lexicalcategories (e.g., a green target among blue distractors) thanwhen target and distractors were from the same lexical category(e.g., a particular green among distractors of a different green),but only when the target was exposed in the right visual field(RVF). Because the RVF projects to the left cerebral hemi-sphere, the dominant hemisphere for language in most adults,and because the effect was eliminated by a concurrent taskoccupying verbal processing resources but not by an equallydifficult task occupying nonverbal resources, the RVF CP find-ing suggests that the spontaneous use of lexical codes in the lefthemisphere may be the origin of the differential visual hemifield

response to colors. A subsequent study (9) with different tasksextended this result and showed stronger category effects (i.e.,faster responses to between-category color pairs than to within-category color pairs) in the RVF than in the left visual field(LVF), although the LVF did show a significant, if weaker,category effect. A third study (12), testing a color term boundaryin Korean that does not exist in English, found CP only in theRVF for relatively rapidly responding subjects but CP in bothvisual fields for slowly responding subjects and no CP at theKorean-only boundary for English-speaking subjects. The au-thors of that study suggest that LVF color CP in slower-responding adults probably reflects cross-callosal transfer; thesame conclusion has been drawn elsewhere (14, 23). Hence it ispossible that in normal adults, color CP is restricted to the lefthemisphere, with apparent LVF CP an artifact of transcallosaltransfer and/or scanning.

Despite growing behavioral evidence for hemifield-dependentcategory effects, the neural correlates of these effects remainunknown. One previous functional MRI (fMRI) study (24)found that, in comparison with hard-to-name colors, perceptualdiscrimination of easy-to-name colors evoked stronger activationin the posterior temporoparietal regions responsible for success-ful word-finding processes, but the study was not designed to lookinto neural substrates of the behavioral RVF superiority in colorperception, and it did not clarify whether linguistic informationaids in the activity of brain regions responsible for color vision.

In the current rapid event-related fMRI study, we investigatedneural mechanisms underlying hemifield-modulated Whorfianeffects in adults. We scanned subjects’ brain activity while theyperformed the visual search task used in the original lateralizedWhorf study (7). The search included colors selected from a setof 4 (Fig. 1A). These 4 colors form a graded series from greento blue, with the green�blue boundary falling between G2 andB1. In the visual search task, each stimulus display consisted ofa ring of colored squares surrounding a central fixation marker(Fig. 1B). Except the target, all the squares were of the samecolor. The target and distractor colors were either from withinthe same lexical category (e.g., a blue target and distractors ofa different shade of blue, ‘‘within category’’) or from differentlexical categories (e.g., a green target and blue distractors,‘‘between category’’). On each trial, participants were asked toindicate whether the target was on the left or right side of thecircle by making timed button-press responses with the corre-sponding hand. In this manner, 2 variables were manipulated:

Author contributions: W.T.S., P.K., W.S.Y.W., A.H.D.C., L.C., K.-K.L., and L.H.T. designedresearch; W.T.S., A.H.D.C., and L.H.T. performed research; W.T.S., A.H.D.C., and L.H.T.analyzed data; and W.T.S., P.K., A.H.D.C., and L.H.T. wrote the paper.

Conflict of interest: The authors declare no conflict of interest. .

1To whom correspondence should be addressed. E-mail: [email protected] [email protected].

This article contains supporting information online at www.pnas.org/cgi/content/full/0903627106/DCSupplemental.

www.pnas.org�cgi�doi�10.1073�pnas.0903627106 PNAS Early Edition � 1 of 6

PSYC

HO

LOG

YN

EURO

SCIE

NCE

the visual field of the target (LVF vs. RVF) and the categoricalrelationship between the target and distractor colors (between-category vs. within-category). There were 2 types of target-distractor pairs: 1-step within-category (G1G2 and B1B2) and1-step between-category (G2,B1).

We tested 2 predictions of the Whorf hypothesis. First, iflexical codes of colors are accessed during visual search, dis-crimination of colors should evoke activations of cortical regionscontributing to language processes, such as left temporoparietalareas and the left inferior prefrontal gyrus. Furthermore, activ-ity levels of these language regions should be stronger forbetween-category than within-category stimuli, especially in theRVF, as predicted by previous behavioral studies (7, 9, 12).Second, if lexical information enhances the perceptual differ-ence rather than merely being accessed as a byproduct of coloridentification, activations of brain regions for color perception,such as visual area 2/3 (V2/3) and visual area 4 (V4), should bealtered by the activation of linguistic information, particularly inthe RVF condition.

ResultsBehavior. Trials in which the participant pressed the wrong key orin which the reaction time (RT) was � 2 SD from the grand meanwere excluded. Two participants’ behavioral data were dis-carded, 1 because of head motion during the brain scan and theother because button responses were recorded inaccurately. Asillustrated in Fig. 1C, with regard to main effects, between-category RTs were significantly faster than within-category RTs[468.80 ms vs. 507.89 ms, F (1, 13) � 27.24, P � 0.001], and RVFRTs were faster than LVF RTs at a level approaching signifi-cance [481.6 ms vs. 495.09 ms, F (1, 13) � 3.41, P � 0.088]. Theinteraction of the 2 variables also approached significance [F (1,13) � 3.62, P � 0.079], with RVF between-category RTs beingthe shortest. For between-category pairs, RVF RTs were signif-icantly faster than LVF RTs (458.9 ms vs. 478.69 ms, t � 2.73, P �0.05). For within-category pairs, LVF RTs were faster by a scant7 ms, not approaching significance (511.497 ms vs. 504.29 ms, t �0.08, P � 0.423). For RVF targets, RTs in the between-categorycondition were 45 ms faster than in the within-category condi-tion (t � 5.68, P � 0.001). For LVF targets, RTs in thebetween-category condition were 33 ms faster than in thewithin-category condition (t � 3.914, P � 0.005). In general, thispattern of behavioral data is consistent with previous studiesusing the same (7) or similar (9, 12) paradigms, suggesting thatthe color CP effects for normal language users are stronger in theRVF than the LVF (i.e., lateralized Whorf).

fMRI Results. We first calculated an average effect of colorperception tasks by collapsing and contrasting all of the colorconditions (LVF within-category, LVF between-category, RVFwithin-category, and RVF between-category) against an implicitbaseline available in the fast event-related fMRI design (Fig. 2and Table 1). Consistent with previous neuroimaging studies ofcolor vision (24–31), subjects showed strong activations in theneural circuitry attributed to color perception, including V2/3and V4 bilaterally. The left temporoparietal areas known tomediate lexical processes were activated also. Bilateral inferiorparietal cortex and motor cortex also showed strong activity,presumably because of motor responses required by the visualsearch task.

The main effect of categorical relationship (between-categoryversus within-category pairs) was computed by collapsing thedata from the 2 visual fields. As depicted in Fig. 3 (Table 2),several language areas involving the left posterior temporopa-rietal region [Brodmann areas (BA) 40 and 39], the left middle-superior temporal gyrus (BA 21 and 22), and the left inferior

C

G1 G2 B1 B2

A

“green” “blue”

400

420

440

460

480

500

520

540

LVF

RVF

Between-category Within-category

*

B

+1

2

3

4

Fig. 1. Experimental materials and behavioral results. (A) Printed-rendered versions of the 4 colors used. (B) Sample display for the visual search task. The targetoccupied any of the 4 positions (position 1, 2, 3, or 4). This example shows a between-category, LVF pair. (C) Behavioral performance in the 4 conditions. Errorbars indicate SEM. *, significant difference in response (P � 0.05).

A

18 24 32 40

212-8-61-

z

0

5

6

B

Fig. 2. An average effect map of color discrimination tasks. Data from all ofthe color conditions (LVF within-category, LVF between-category, RVF within-category, and RVF between-category) were collapsed. (A) Lateral view. (B)Axial sections. The significance threshold is P � 0.05 FDR-corrected. L, lefthemisphere; R, right hemisphere.

2 of 6 � www.pnas.org�cgi�doi�10.1073�pnas.0903627106 Siok et al.

prefrontal gyrus (BA 47) were strongly activated. These regionshad been shown to govern lexical search and semantic retrievalin past lesion and neuroimaging investigations of aphasia andlanguage functions (24, 32–47); their activation in the visualsearch task indicates that linguistic information of colors israpidly activated and represented in the brain.

To ascertain whether there is a stronger activation categoryeffect in the RVF than in the LVF, we calculated separateactivation maps for each visual field, relating between-categorycolor discrimination to within-category color discrimination,with a significance level for between-condition differences beingset at P � 0.05 false discovery rate (FDR) corrected for multiplecomparisons. For the RVF color stimuli, activation produced bydiscrimination of colors from different lexical categories minusactivation from same-category colors was very strong in the leftposterior temporoparietal region (BA 40), the left middle tem-poral gyrus (BA 21), and the left inferior prefrontal cortex at BA47 (Fig. 4 A and B). This pattern of data converges with theaforementioned results from the main effect of categoricalrelationship. The total activation volume in these 3 regions, asindexed by number of voxels, is 1007 (Fig. 4D). Equally impor-

tant is the significant difference in response delay between colorsin the same lexical category and colors in differing lexicalcategories, as illustrated in an averaged response delay differ-ence map (Fig. 4C). On a voxelwise basis, hemodynamic re-sponses were slower in all 3 language regions for same-categorypairs than for different-category pairs, suggesting that lexicalinformation speeds up the perceptual processing of the RVFcolors.

Nonetheless, when the color stimuli were displayed in theLVF, discrimination of colors from different lexical categoriesminus colors from the same lexical category did not provokestronger activation of any of language-related regions such as theleft posterior temporoparietal network (BA 40) when the sig-nificance threshold was set at P � 0.05 FDR corrected. When aless stringent threshold of P � 0.005 uncorrected was used, theactivation of the left posterior temporoparietal regions was seen,with only 92 voxels totally (see Fig. S1). In addition, differencesin mean hemodynamic delays were not found in this neuralcircuitry. These results indicate that differences between theactivation of language regions by the LVF between-categorystimuli and the LVF within-category stimuli, if any, would bevery weak. This finding confirms previous findings that LVFstimuli may activate left-hemisphere language areas by virtue ofa longer and ‘‘noisier’’ transcallosal pathway (48, 49).

To determine whether lexical color categories are used tosharpen the perceptual difference through enhanced activationof brain regions for color perception, particularly for RVF colorstimuli, we performed a whole-brain, voxel-based analysis of theinteraction between visual hemifield and categorical relation. Asmall set of regions hypothesized a priori to be involved in colorperception on the basis of prior results (24) as well as the lexicalcategory effect map of this study were defined to determine thesignificance of predicted peaks. These regions included the leftvisual areas (V2/3 and V4) and the left language regions. Peaksthat survived the whole-brain analysis thresholded at P � 0.005(uncorrected) and small volume correction with P � 0.05FDR-corrected were considered significant. Relevant regionsemerging from this analysis are the left temporoparietal area(BA 40) responsible for language processes and V2/3 crucial forcolor vision. Fig. 5 depicts averaged activity levels in the 4conditions. The result shows that activity levels in both V2/3 andBA 40 were significantly enhanced when colors from differentlexical categories were exposed in the RVF. Thus, it seems thatlexical category information enhances the neuronal response atV2/3 for colors appearing in the RVF.

R

L

A

21 31

-13 -7z

0

4.5B

Fig. 3. A main effect map of categorical relationship (between-category andwithin-category colors). Data from the 2 visual hemifields were collapsed. (A)Lateral view. (B) Axial sections. The significance threshold is P � 0.05 FDR-corrected. L, left hemisphere; R, right hemisphere.

Table 1. Coordinates of activation peaks: An average effectof color discrimination tasks

Regions activatedBrodmann

area

Coordinates

Z-ScoreX Y Z

OccipitalLeft V4 �28 �69 �12 5.71Left V4 alpha �36 �50 �19 6.60

�46 �59 �17 6.31Right V4 26 �69 �13 6.05Right V4 alpha 46 �59 �19 5.77V1 4 �72 4 5.78

0 �72 �1 5.53V2/3 �28 �84 21 6.60

28 �86 25 5.84�16 �64 9 5.54

FrontalLeft inferior frontal gyrus 44 �44 5 27 3.88Left middle frontal gyrus 6 �36 �3 57 5.09

9 �53 9 33 3.1010 �30 53 19 3.65

Left precentral gyrus 4 �51 �13 52 5.166 �32 �18 64 5.16

Left postcentral gyrus 2 �48 �25 44 5.39Right inferior frontal gyrus 44 48 11 29 4.42Right precentral gyrus 4 48 �7 57 5.35

6 40 �3 59 4.934 53 �13 47 4.856 42 �1 11 3.99

Cingulate gyrus 24 �4 2 46 6.1332 �2 17 36 4.98

Left insula �32 16 5 3.28ParietalLeft superior parietal lobule 7 �26 �58 51 4.97Right superior parietal lobule 7 26 �56 45 5.15TemporalLeft superior temporal gyrus 22 �57 6 �2 4.99

42 �61 �17 14 4.98Right superior temporal gyrus 22 63 0 4 3.71

38 55 17 �8 3.1322 61 10 1 3.05

Subcortical areasThalamus �12 �19 6 5.18

16 �17 10 3.94

Siok et al. PNAS Early Edition � 3 of 6

PSYC

HO

LOG

YN

EURO

SCIE

NCE

DiscussionWe have found brain language regions participating in categor-ical color perception when subjects performed a visual searchtask. The activity of the language regions, however, was modu-lated by the visual field in which the stimulus appeared, asdemonstrated by the following findings. First, in the RVF,perception of target and distractor colors from different linguis-tic categories (contrasted with target and distractor colors fromthe same linguistic category) activates language areas includingthe posterior temporoparietal region, the middle temporal gy-rus, and the inferior prefrontal cortex in the left cerebralhemisphere, but in the LVF perception of target and distractorcolors from different lexical categories is not associated withstronger activity in any language regions. Second, the activationof language regions seems to exhibit a slower hemodynamicresponse for colors from the same lexical category than forcolors from different lexical categories, but only when the colorsare presented in the RVF.

These findings therefore extend prior neuropsychological andbrain mapping studies (7, 9, 12, 24) and unequivocally demon-

strate hemifield-dependent activations of language regions in acolor-discrimination task.

Lexical color information not only was accessed in colordiscrimination but also enhanced the activation of color regionV2/3. When the colors exposed in the RVF were from differinglexical categories, activation of V2/3 was much stronger thanwith other color conditions. Notably, the increased activity ofV2/3 for the RVF between-category colors coincided with theincreased activity of the posterior temporoparietal region forlanguage processes, as demonstrated by the significant interactionof visual field and categorical relation, suggesting that CP of colorprovokes orchestrated cortical activity occurring within subsystemsinvolving the posterior temporoparietal region and V2/3.

We tentatively infer that the posterior temporoparietal cortexserves as a top-down control source that interacts with andmodulates the activity of the visual cortex (V2/3) serving data-driven analysis of visual stimuli. Anatomical studies have foundmultiple reciprocal neural pathways between the parietal cortexand visual processing areas, and these pathways may govern suchcontrol (50–53). Our results are consistent with lesion studies of

Talariachcoordinates

3.4430-55-4817740L supramarginal gyrus

4.132-41-6741021L middle temporal gyrus

5.88-1528-4442047L inferior frontal gyrus

Z value

zyx

VoxelBABrain Region

D

A

B

C

Fig. 4. Brain regions with significant activation during the identification ofcolors from different lexical categories in the right visual field in comparisonwith colors from the same lexical category in the right visual field. (A) Lateralview. (B) Language regions in the brain showing stronger activation in thebetween-category condition than in the within-category condition. (C) Lan-guage regions in the brain that exhibited significantly slower hemodynamicresponses in the within-category color condition than in the cross-categorycolor condition. (D) Coordinates of activation peaks in the 3 language areas.The significance threshold is P � 0.05 FDR-corrected.

Table 2. Coordinates of activation peaks: Main effectof categorical relationship

Regions activatedBrodmann

area

Coordinates

Z scoreX Y Z

FrontalLeft inferior frontal gyrus 47 �32 21 �1 5.97

47 �44 28 �13 4.3344 �42 5 22 5.2745 �44 17 19 4.41

Left middle frontal gyrus 8 �38 31 43 3.09Left superior frontal gyrus 8 �12 33 48 3.74Left precentral gyrus 6 �38 0 37 4.74Left insula �40 2 2 3.38Right inferior frontal gyrus 47 34 21 �3 6.06Right middle frontal gyrus 46 42 40 16 3.58Right superior frontal gyrus 8 12 32 52 4.40Medial frontal 11 �2 34 �19 4.66

6 �6 �5 61 3.68Cingulate gyrus 32 6 23 36 4.62ParietalLeft inferior parietal lobule 40 �40 �43 39 4.77Left precuneus 7 �20 �62 36 5.82Right inferior parietal lobule 40 44 �35 44 4.00Right superior parietal lobule 7 28 �56 49 4.44Right precuneus 7 12 �67 49 3.53Right supramarginal gyrus 40 46 �49 25 3.38TemporalLeft inferior temporal gyrus 20 �51 �13 �30 3.31Left middle temporal gyrus 21 �67 �29 �7 3.86Left superior temporal gyrus 22 �65 �46 21 3.26Left fusiform gyrus 37 �48 �49 �11 3.85Left angular gyrus 39 �44 �64 31 2.92Right inferior temporal gyrus 20 53 �9 �25 3.49Right superior temporal gyrus 38 36 22 �21 4.31Right fusiform gyrus 37 46 �55 �7 4.47Right angular gyrus 39 51 �67 29 3.44OccipitalV2/3 18 8 �85 13 3.98

18 4 �72 28 3.6118 �8 �83 13 3.1319 4 �84 32 2.75

Right superior occipital gyrus 19 32 �69 26 3.71Limbic lobePosterior cingulate 31 6 �61 14 3.32

4 of 6 � www.pnas.org�cgi�doi�10.1073�pnas.0903627106 Siok et al.

visual attention (54) indicating that the posterior parietal cortexinteracts with the response of neurons in the visual areas in waysthat may fundamentally influence object representations.

At present a direct functional connection and interactionbetween the posterior temporoparietal region and the colorregion(s) in color perception has not been established. Futureresearch may address this question by performing an effectiveconnectivity analysis of fMRI data and/or by providing morerefined time-course information with the event-related potentialtechnique. Our study nevertheless has identified the neuralcorrelates of the behavioral RVF advantage in color discrimi-nation and thus shed light on the mechanisms that underlieWhorfian effects. Language, by enhancing the activation level ofthe visual cortex, differentially influences the discrimination ofcolors presented in the left and right visual hemifields.

Materials and MethodsSubjects. Beijing college students [8 males and 8 females; mean age, 23.7 years(SD 1.8 years)] participated in the fMRI experiment. The data of 1 subject werediscarded because of head motion and a low identification score in the colorboundary test. Subjects were paid for their participation and gave informedconsent according to guidelines set by the Administrative Panels on HumanSubjects in Medical Research of the Beijing MRI Center for Brain Research atthe Chinese Academy of Sciences. They were tested with the Ishihara test forcolor blindness; all subjects had normal color vision and no history of neuro-logical or psychiatric illness. All subjects were strongly right-handed.

Stimuli and Experimental Design. The RGB values of the 4 colors were as follows(see Fig. 1A): G1 � 0, 171, 129; G2 � 0, 170, 149; B1 � 0, 170, 170; B2 � 0, 149,

170. The brightness and saturation values were adjusted to make them equal,based on the independent judgments of 4 observers. The RGB values for thebackground were 210, 210, and 210. CIEL*u*v* values are given in Table 3. Theinter-pair distances are (G1,G2) � 16.3 �E, (G2,B1) � 17.48 �E, and (B1,B2) �

19.47 �E. The mean within-category distance, 17.89 �E, slightly exceeds thebetween-category (G2,B1) distance, 17.48 �E .

A rapid event-related design was used. During each trial, a ring of 12colored squares surrounding the fixation marker was presented simulta-neously for 200 ms against a gray background (Fig. 1B), followed by a fixationscreen against a gray background. Subjects indicated whether the target wason the left or right side of the circle by making button-press responses with thecorresponding hand as quickly and as accurately as possible. The duration ofthe fixation screen varied to jitter the blood oxygen level-dependent (BOLD)responses. Inter-stimulus intervals of 1800, 2800, or 3800 ms were assignedrandomly to the trials, resulting in corresponding stimulus onset asynchroniesof 2000, 3000, and 4000 ms. There were 6 target–distractor pairs formed byusing all 1-step pairwise combinations of the 4 colors (3 pairs: G1G2, B1B2, andG2B1) and having each member of a pair serve once as target and once asdistractor. The target occupied any of the 4 positions (position 1, 2, 3, or 4 inFig. 1B), and there were 24 possible stimulus configurations. There were 400trials in total. In half of the trials, the target was located to the left of center(position 1 or 2), and in the other half of the trials it was located to the rightof center (position 3 or 4). In addition, half of the trials presented within-category combinations (G1G2 or B1B2), and the other half presented thebetween-category combination (G2B1).

The stimuli were presented via a liquid crystal display projector and wereback-projected onto a projection screen placed at the end of the scanner bore.Subjects viewed the rear projection screen through a mirror attached to the headcoil. The distance from the projection screen to the mirror was 70 cm, and thedistance from the mirror to the eyes of the subject was 10 cm. The inner edge ofthe target color was presented 3.9° to the right or to the left of a centrallypresented ‘‘�’’. Hence, the stimuli were separated by a visual angle of 7.8°.

After the fMRI scans, subjects were given a blue–green lexical boundarytest. On each trial, a square stimulus (1 of the colors, G1, G2, B1, or B2) waspresented centrally on a gray background for 200 ms, followed by an 1800-msinterval. Participants indicated whether the stimulus was green or blue bypressing 1 of 2 keys, corresponding to the Mandarin Chinese words for‘‘green’’ and ‘‘blue,’’ respectively. Each stimulus was presented 10 times in atotal of 40 randomized trials. Fifteen subjects identified more than 93% of thepresentations of G1 and G2 as ‘‘green’’ and of B1 and B2 as ‘‘blue.’’ One subjectidentified only 53% of the presentations in this way; the data from this subjectwere discarded.

Image Acquisition and Data Analysis. Details of image acquisition and dataanalysis are given in the SI text.

ACKNOWLEDGMENTS. We thank Liu Haiqi, Zhou Ke, Wei Zhou, Joey Li, andLiu Zhendong for help with the experiments. This research was supported bya 973 grant from the National Strategic Basic Research Program of the Ministryof Science and Technology of China (2005CB522802), the Knowledge Innova-tion Program of the Chinese Academy of Sciences, the University of HongKong, Grant 811-5020 from the Shun Hing Institute of Advanced Engineeringof Chinese University of Hong Kong, and by Grant 0418404 from the U.S.National Science Foundation.

1. Carroll JB (1956) Language, Thought, and Reality: Selected Writings of Benjamin LeeWhorf (MIT, Cambridge, MA).

2. Kay P, Kempton W (1984) What is the Sapir-Whorf hypothesis? Am Anthropol 86:65–79.3. Gilbert AL, Regier T, Kay P, Ivry RB (2008) Support for lateralization of the Whorf effect

beyond the realm of color discrimination. Brain and Language 105:91–98.4. Gumperz JJ, Levinson SC (1996) Rethinking Linguistic Relativity (Cambridge Univ Press,

Cambridge, UK).5. Casasanto D (2005) Crying ‘‘Whorf’’. Science 307:1721–1722.6. Gordon P (2005) Response. Science 307:1722.

7. Gilbert AL, Regier T, Kay P, Ivry RB (2006) Whorf hypothesis is supported in the rightvisual field but not the left. Proc Natl Acad Sci USA 103:489–494.

8. Winawer J, et al. (2007) Russian blues reveal effects of language on color discrimina-tion. Proc Natl Acad Sci USA 104:7780–7785.

9. Drivonikou GV, et al. (2007) Further evidence that Whorfian effects are stronger in theright visual field than the left. Proc Natl Acad Sci USA 104:1097–1102.

10. Butterworth B, Reeve R, Reynolds F, Lloyd D (2008) Numerical thought with andwithout words: Evidence from indigenous Australian children. Proc Natl Acad Sci USA105:13179–13184.

-0.16

-0.12

-0.08

-0.04

0.00

0.04

0.08

% B

old

Sig

nal

Ch

an

ge

BA 40

-0.10

-0.05

0.00

0.05

0.10

LB LW RB RW

V2/3

% B

old

Sig

nal

Ch

an

ge

Fig. 5. Regions of interest that survive small volume correction with P � 0.05FDR-corrected and a whole-brain voxel-based analysis (thresholded at P �0.005 uncorrected for multiple corrections) of the interaction between visualfield and categorical relationship. LB, left visual field between-category; LW,left visual field within-category; RB, right visual field between-category; RW,right visual field within-category.

Table 3. CIEL*u*v* values and inter-pair distances

Stimulus L* u* v* Pair �E

G1 62.263 �52.327 23.044G2 62.44 �50.447 6.856 (G1,G2) 16.29776282B1 63.054 �48.768 �10.53 (G2,B1) 17.47767241B2 56.483 �41.453 �27.34 (B1,B2) 19.47468526

Siok et al. PNAS Early Edition � 5 of 6

PSYC

HO

LOG

YN

EURO

SCIE

NCE

11. Roberson D, Davidoff J (2000) The categorical perception of colors and facial expres-sions: The effect of verbal interference. Memory & Cognition 28:977–986.

12. Roberson D, Pak HS, Hanley JR (2008) Categorical perception of colour in the left andright visual field is verbally mediated: Evidence from Korean. Cognition 107:752–762.

13. Heider ER, Olivier DC (1972) The structure of the color space in naming and memory fortwo languages. Cognit Psychol 3:337–354.

14. Kay P, Regier T, Gilbert AL, Ivry RB. (in press) Lateralized Whorf: Language influencesperceptual decision in the right visual field. Language, Evolution, and the Brain, edsMinett JW, Wang W.S-Y. (Hong Kong, The City Univ of Hong Kong Press).

15. Franklin A, Clifford A, Williamson E, Davies I (2005) Color term knowledge does notaffect categorical perception of color in toddlers. Journal of Experimental ChildPsychology 90:114–141.

16. Pinker S (1994) The Language Instinct (Morrow, New York).17. Bloom AH (1981) The Linguistic Shaping of Thought: A Study in the Impact of

Language on Thinking in China and the West (Lawrence Erlbaum Associates, Hillsdale,NJ).

18. Au TK (1983) Chinese and English counterfactuals: The Sapir-Whorf hypothesis revis-ited. Cognition 15:155–187.

19. Li P, Gleitman L (2002) Turning the tables: Language and spatial reasoning. Cognition83:265–294.

20. Levinson SC, Kita S, Haun DB, Rasch BH (2002) Returning the tables: Language affectsspatial reasoning. Cognition 84:155–188.

21. Heider ER (1972) Universals in color naming and memory. J Exp Psychol 93:10–20.22. Kay P, Regier T (2006) Language, thought and color: Recent developments. Trends in

Cognitive Sciences 10:51–54.23. Franklin A, et al. (2008) Categorical perception of color is lateralized to the right

hemisphere in infants, but to the left hemisphere in adults. Proc Natl Acad Sci USA105:3221–3225.

24. Tan LH, et al. (2008) Language affects patterns of brain activation associated withperceptual decision. Proc Natl Acad Sci USA 105:4004–4009.

25. Martin A, Haxby JV, Lalonde FM, Wiggs CL, Ungerleider LG (1995) Discrete cortical regionsassociated with knowledge of color and knowledge of action. Science 270:102–105.

26. Zeki S, et al. (1991) A direct demonstration of functional specialization in human visualcortex. J Neurosci 11:641–649.

27. Wandell BA (1999) Computational neuroimaging of human visual cortex. Annu RevNeurosci 22:145–173.

28. Wade AR, Brewer AA, Rieger JW, Wandell BA (2002) Functional measurements ofhuman ventral occipital cortex: Retinotopy and colour. Philos Trans R Soc London B357:963–973.

29. Bartels A, Zeki S (2004) Functional brain mapping during free viewing of natural scenes.Hum Brain Mapp 21:75–85.

30. Howard RJ, et al. (1998) The functional anatomy of imagining and perceiving color.NeuroReport 9:1019–1023.

31. Dougherty RF, et al. (2003) Visual field representations and locations of visual areasV1/2/3 in human visual cortex. Journal of Vision 3:586–598.

32. Gorno-Tempini ML, et al. (2004) Cognition and anatomy in three variants of primaryprogressive aphasia. Ann Neurol 55:335–346.

33. Sonty SP, et al. (2003) Primary progressive aphasia: PPA and the language network. AnnNeurol 53:35–49.

34. Oxbury JM, Oxbury SM, Humphrey NK (1969) Varieties of colour anomia. Brain 92:847–860.

35. Damasio AR, Damasio H (1983) The anatomic basis of pure alexia. Neurology 33:1573–1583.

36. Davidoff JB, Ostergaard AL (1984) Colour anomia resulting from weakened short-termcolour memory. A case study. Brain 107:415–431.

37. Gabrieli JD, Poldrack RA, Desmond JE (1998) The role of left prefrontal cortex inlanguage and memory. Proc Natl Acad Sci USA 95:906–913.

38. Hoeft F, et al. (2007) Functional and morphometric brain dissociation between dyslexiaand reading ability. Proc Natl Acad Sci USA 104:4234–4239.

39. Demb JB, et al. (1995) Semantic encoding and retrieval in the left inferior prefrontalcortex: A functional MRI study of task difficulty and process specificity. J Neurosci15:5870–5878.

40. Poldrack RA, et al. (1999) Functional specialization for semantic and phonologicalprocessing in the left inferior prefrontal cortex. NeuroImage 10:15–35.

41. Schlaggar BL, McCandliss BD (2007) Development of neural systems for reading. AnnuRev Neurosci 30:475–503.

42. Bokde AL, Tagamets MA, Friedman RB, Horwitz B (2001) Functional interactions of theinferior frontal cortex during the processing of words and word-like stimuli. Neuron30:609–617.

43. Bookheimer S (2002) Functional MRI of language: New approaches to understandingthe cortical organization of semantic processing. Annu Rev Neurosci 25:151–188.

44. Goswami U (2006) Neuroscience and education: From research to practice? NatureReviews Neuroscience 7:406–413.

45. Price CJ (2000) The anatomy of language: Contributions from functional neuroimag-ing. J Anat 197:335–359.

46. Eden G, Moats L (2002) The role of neuroscience in the remediation of students withdyslexia. Nature Neuroscience 5:1080–1084.

47. Wagner AD, Pare-Blagoev EJ, Clark J, Poldrack RA (2001) Recovering meaning: Leftprefrontal cortex guides controlled semantic retrieval. Neuron 31:329–338.

48. Grusser OJ, Landis T (1991) Visual Agnosias and Other Disturbances of Visual Percep-tion and Cognition (London, Macmillan Press).

49. Kinsbourne M (1970) The cerebral basis of lateral asymmetries in attention. ActaPsychologica 33:193–201.

50. Cavada C, Goldman-Rakic P (1989) Posterior parietal cortex in rhesus monkey: I.Parcellation of areas based on distinctive limbic and sensory corticocortical connec-tions. J Comp Neurol 287:393–421.

51. Cavada C, Goldman-Rakic P (1991) Topographic segregation of corticostriatal projec-tions from posterior parietal subdivisions in the macaque monkey. Neuroscience42:683–696.

52. Distler C, Boussaoud D, Desimone R, Ungerleider LG (1993) Cortical connections ofinferior temporal area TEO in macaque monkeys. J Comp Neurol 334:125–150.

53. Webster MJ, Bachevalier J, Ungerleider LG (1994) Connections of inferior temporalareas TEO and TE with parietal and frontal cortex in Macaque monkeys. Cereb Cortex4:470–483.

54. Friedman-Hill SR, Robertson LC, Desimone R, Ungerleider LG (2003) Posterior parietalcortex and the filtering of distractors. Proc Natl Acad Sci USA 100:4263–4268.

6 of 6 � www.pnas.org�cgi�doi�10.1073�pnas.0903627106 Siok et al.

All Those Seeing Color, Say Eye!Student Sheet

Name:_____________________________________

Part IGo to An Eye on Color slideshow (http://www.thetech.org/exhibits_events/online/color/vision/),on the Tech Museum of Innovation website. First, click on the link titled Inside Your Eyeball.Then navigate through the slideshow by clicking on the link within the text of each slide. Stopwhen the slideshow is over. As you read the slides, answer the following questions:

1. What happens when waves of light enter your eye? Discuss the role of the pupil, retina, andoptic nerve.

2. Where are rods and cones located?

3. What are rods and cones? What do they allow you to see?

4. In what area of the retina are cones concentrated?

5. When do cones work best?

6. In what area of the retina are rods concentrated?

7. When do rods work best?

8. You’ve just read what happens in the eye: light goes through the pupil, then hits the rodsand cones and causes a chemical reaction. What understands this reaction and carries themessage to the brain?

9. What causes the blind spot?

10. Why do certain animals (like the owl and bee) see colors differently than humans?

Wait for your teacher to discuss what you just learned.

Part IIGo to A Big Look at the Eye on the KidsHealth website.http://www.kidshealth.org/kid/body/eye_SW.html

Navigate through the article by clicking on the “Next Page” link at the bottom of each slide. Afteryou read the “The Eyes Take the Prize” page, click on the link “Want to see the eye in action?”Then click on “The Eye.” As you read the article, answer the following questions:

1. What are functions of the eyelid and blinking?

2. What is a function of eyelashes?

3. Draw a diagram of the eye labeling the sclera, cornea, iris, and pupil.

Science NetLinks Student Sheet-All Those Seeing Color, Say Eye!All rights reserved. Science NetLinks Student Sheets may be reproduced for educational purposes.

4. What part of the eye lets light in?

5. What happens to the pupil as you enter a dark room? Why?

6. Once light enters through the pupil, what part of the eye focuses light on the retina?

7. What carries messages from the eye to the brain?

Color Constancy based on the Grey-Edge Hypothesis

J. van de Weijer Th. Gevers

Intelligent Sensory Information SystemsFaculty of Science, University of Amsterdam

Kruislaan 403, 1098 SJ Amsterdam, The Netherlands{joostw, gevers}@science.uva.nl

Abstract

A well-known color constancy method is based on the Grey-World assumption i.e. the average reflectance of surfacesin the world is achromatic. In this article we propose anew hypothesis for color constancy, namely the Grey-Edgehypothesis assuming that the average edge difference in ascene is achromatic. Based on this hypothesis, we proposean algorithm for color constancy.

Recently, the Grey-World hypothesis and the max-RGBmethod were shown to be two instantiations of a Minkowskinorm based color constancy method. Similarly we alsopropose a more generale version of the Grey-Edge hypoth-esis which assumes that the Minkowsky norm of deriva-tives of the reflectance of surfaces is achromatic. The al-gorithms are tested on a large data set of images under dif-ferent illuminants, and the results show that the new methodoutperforms the Grey-World assumption and the max-RGBmethod. Results are comparable to more elaborate algo-rithms, however at lower computational costs.

1 Introduction

Color constancy is the ability to recognize colors of objectsinvariant of the color of the light source [1], [2] [6]. It gen-erally consists of two steps. Firstly, the light source coloris estimated from the image data. Secondly, illuminant in-variant descriptors are computed, which is usually done byadjusting the image for the color of the light source suchthat the object colors resemble the colors of the objects un-der a known light source.

A simple color constancy method, called max-RGB, esti-mates the light source color from the maximum response ofthe different color channels [1]. Another well-known colorconstancy method is based on the Grey-World hypothesis[4], which assumes that the average reflectance in the sceneis achromatic. Although more elaborate algorithms exists,methods like Grey-World and max-RGB are still widelyused because of their low computational costs.

Recently, Finlayson and Trezzi [5] showed that the max-RGB method and the Grey-World method can be interpretedas the same algorithm applied with different instantiationsof the error function. The max-RGB method is shown tobe equal to applying the L∞ Minkowski norm and Grey-World is equal to using the L1 norm. They further showthat the best color constancy results are attained with the L6

norm. Although these simple color constancy algorithmsare slightly outperformed by more elaborate methods, e.g.color gamut mapping (for an overview see [1] [2]), they per-form surprisingly well while their computational costs aresignificantly lower.

In this paper, we pursue color constancy by the Grey-Edge hypothesis, which assumes the average edge differ-ence in the scene to be achromatic. The method is based onthe observation that the distribution of color derivatives ex-hibit the largest variation in the light source direction. Theaverage of these derivatives is used to approximate this di-rection. The method is tested on a large database of colorfulobjects under varying lighting conditions and different illu-minants. We further extend the method similarly to [5] andalso derive color constancy for the error based on the vari-ous Minkowski norms.

The paper is organized as follows. In section 2 colorconstancy based on the Grey-World hypothesis is explained.In section 3 we propose the Grey-Edge hypothesis for colorconstancy computation. Section 4 contains experiments andSection 5 finishes with concluding remarks.

2 The Grey-World Hypothesis

The image values, f = (R,G,B)T , for a Lambertian sur-

face are dependent on the light source e (λ), where λ isthe wavelength, the surface reflectance s (λ) and the camerasensitivity functions c (λ) = (R (λ) , G (λ) , B (λ))

f =

∫

ω

e (λ) s (λ) c (λ) dλ, (1)

0-7803-9134-9/05/$20.00 ©2005 IEEE II-722

O1O1

O1

O2O2

O3

O2

O3O3

Figure 1: Three acquisitions of the same scene under different light sources [3]. On the bottom line the derivative distribu-tions, where the axes are the opponent color derivatives and the surfaces indicate derivative values with equal occurrenceand darker surfaces indicating a more dense distribution. Note the shift of the orientation of the distribution of the derivativeswith the changing of the light source.

where ω is the visible spectrum and bold fonts are appliedfor vectors. The goal of color constancy is to estimatethe light source color e (λ), or its projection on the RGB-kernels,

e =

Re

Ge

Be

=

∫

ω

e (λ) c (λ) dλ, (2)

given the image values f (x), where x is the spatial coordi-nate in the image. The task of color constancy is not attain-able without further assumptions.

Buchsbaum [4] proposes the Grey-World hypothesis,which assumes that the average reflectance in a scene isachromatic:

∫

s (λ,x) dx∫

dx= k. (3)

The light source color can now be estimated by computingthe average pixel value , since

∫

f(x)dx∫

dx= 1

∫

dx

∫ ∫

ω

e (λ) s (λ,x) c (λ) dλdx

= k∫

ω

e (λ) c (λ) dλ = ke, (4)

which yields the normalized light source color :e =ke/ |ke|. This is indeed a very simple algorithm to findthe light source color of a scene.

In [5] it is shown that the Grey-World hypothesis canbe improved by replacing the averaging operation by theMinkowski norm. In this case Eq. 4 can be rewritten as

(∫

fp (x) dx∫

dx

)1

p

= ke. (5)

For p = 1 the equation is equal to the Grey-World as-sumption. For p = ∞ it is equal to color constancy bymax-RGB, which is based on the assumption that the max-imum response in the channels is caused by a white patch.Hence, the maximum responses yield an estimate of thelight source. Finlayson and Trezzi [5] found that the bestresults are obtained with a Minkowski norm with p = 6.

3 The Grey-Edge Hypothesis

As an alternative to the Grey-World hypothesis, we proposethe Grey-Edge hypothesis; the average of the reflectance

II-723

Figure 2: Examples of the images in group A and B [3].

differences in a scene is achromatic

∫

|sx (λ, x)| dx∫

dx= k. (6)

With the Grey-Edge assumption the light source color canbe computed from the average color derivative in the imagegiven by:

∫

|fx(x)|dx∫

dx= 1

∫

dx

∫ ∫

ω

e (λ) |sx (λ,x)| c (λ) dλdx

= k∫

ω

e (λ) c (λ) dλ = ke,(7)

where |fx (x)| = (|Rx (x)| , |Gx (x)| , |Bx (x)|)T . The

Grey-Edge hypothesis originates from the observation thatthe color derivative distribution of images forms a relativelyregular, ellipsoid-like shape, of which the long axis coin-cides with the light source color. In Fig. 1 the color deriva-tive distribution is depicted for three images. The colorderivatives are rotated to the opponent color space

O1x = Rx−Gx√2

O2x = Rx+Gx−2Bx√6

O3x = Rx+Gx+Bx√3

. (8)

In the opponent color space, O3 coincides with the whitelight direction. For the scene under white light (the left-most picture) the distribution of the derivatives are centeredalong the O3 or white-light axis. Once we change the colorof the light source as in the second and third picture, thedistribution of the color derivatives no longer align with thewhite-light axis. Color constancy based on the Grey-Edgeassumption can be interpreted as skewing the color deriva-tive distribution such that the average derivative is in the O3orientation.

Similarly as for the Grey-World based color constancy,the Grey-Edge hypothesis can also be adapted to incorpo-

rate the Minkowsky norm

(∫

|fx (x)|pdx

∫

dx

)1

p

= ke. (9)

Color constancy based on this equation assumes that the p-th Minkowski norm of the derivative of the reflectance in ascene is achromatic.

4 Experiments

To test the Grey-Edge hypothesis the algorithm is tested ona large data set of colorful object under varying light sources[3]. The data set is split in two groups. Group A consistsof 321 images with varying light sources over a total of 32scenes and group B consists of 220 images of 22 scenes (seeexamples in Fig. 2). For all images the correct light source ismeasured, el. As an error measure we use the angular errorbetween the the estimated light source ee and the measuredlight source el

angular error = cos−1 (el · ee) , (10)

where the (.) indicates the normalized vector. Results ofother color constancy algorithms on this standard data setare available in [2], [7], [5]. For the derivatives Gaussianderivatives with σ = 3 were applied.

In Fig. 3 the results for the Grey-World and the Grey-Edge assumption as a function of the applied norm, p, aredepicted. The results of the Grey-World are taken from[5]. The angular error for the Grey-Edge method outper-forms the Grey-World method for both groups of images.Whereas the Grey-World method finds a minimum error forthe same norm, p = 6 for both groups of images, for theGrey-Edge method the behavior as a function of p variesfor the two groups of images. If we compare p = 6 for theGrey-World with p = 16 for the Grey-Edge based method,

II-724

group A group B

¥¥

Figure 3: Angular error of the Grey-World and the Grey-Edge method as a function of the applied Minkowski norm.

MeanGrey-World (=L1-norm) 9.8Max-RGB (=L∞-norm) 9.2L6-norm Grey-World 6.3L6-norm Grey-Edge 5.7Color by Correlation 9.9Gamut Mapping 5.6GCIE Version 3, 11 lights 4.9

Table 1: Mean angular error (degrees) for various color con-stancy methods on group A images [7].

we attain an improvement of 9% for the images in group Aand of 10 % for the images in group B.

Also the p = ∞ norm, which is the Grey-Edge variant onthe max-RGB method, achieves a good performance. Thelight source is computed from the assumption that the lightsource is equal to the maximum derivatives of the variouscolor channels.

Results of more complex color constancy methods, suchas gamut mapping and color-by-correlation, have been re-ported in [2], [7] for the images in group A. The resultsare comparable to the results reported here and only twomethods perform slightly better, see Table 1. For examplefor Gamut mapping an angular error of 5.6◦ was reported(opposed to 5.7◦ for the Grey-Edge based color constancy).These methods are, however, considerably more complexand therefore require higher computational costs. In con-clusion, the presented Grey-Edge method is an useful alter-native when computational speed is an issue, with a perfor-mance comparable to the best results reported in literature.

5 Conclusions

In this paper we proposed a color constancy algorithm basedon the Grey-Edge hypothesis which assumes the averageedge difference in a scene to be achromatic. Further, anextension based on the Minkowski norm is proposed. Thealgorithm is tested on a large data set and is shown to out-perform color-constancy based on the Grey-World hypoth-esis and the max-RGB assumption.

References[1] K. Barnard, V. Cardei, and B.V. Funt. A comparison of com-

putational color constancy algorithms-part i: Methodologyand experiments with synthesized data. IEEE transactions onImage Processing, 11(9):972–984, September 2002.

[2] K. Barnard, V. Cardei, and B.V. Funt. A comparison of com-putational color constancy algorithms-part ii: Experimentswith image data. IEEE transactions on Image Processing,11(9):985–996, September 2002.

[3] K. Barnard, L. Martin, B.V. Funt, and A. Coath. A data set forcolour research. Color Research and Application, 27(3):147–151, 2002.

[4] G. Buchsbaum. A spatial processor model for object colourperception. Journal of the Franklin Institute, 310, 1980.

[5] G.D. Finlayson and E. Trezzi. Shades of gray and colourconstancy. In IS&T/SID Twelfth Color Imageing Conference,pages 37–41, 2004.

[6] D.A. Forsyth. A novel algorithm for color constancy. Inter-national Journal of Computer Vision, 5(1):5–36, 1990.

[7] S.D. Hordley G.D. Finlayson and I. Tastl. Gamut constrainedilluminant estimation. In Proc. of the Ninth IEEE Interna-tional Conference on Computer Vision, Nice, France, 2003.

II-725

List

Research on color

books

• ‘Josef Albers. Homage to the square’, —, Hochshule fuür Angewandte Kunst Wien,

1992, ISBN 3-85211-018-1

• ‘Josef Albers. Werke auf Papier’, —, Bonn, 1998, ISBN 3-929790-28-9

• ‘Das Farbenmischbuch’, —, Verlag Berliner Union GmbH, 1954

• ‘Elsworth Kelly. In between spaces’, —, Fondation Beyeler, 2002, ISBN 3-905632-21-7

• ‘Elsworth Kelly. The years in France, 1984-1954’, —, National Gallery of Art,

Washington, 1992, ISBN 0-89468-185-0

• ‘Richard Paul Lohse’, —, Stedelijk Van Abbe Museum, Eindhoven, 1971

• ‘Richard Paul Lohse. Color becomes form’, —, Annely Juda Fine Art, London, 1997

• ‘Interaction of Color’, Josef Albers, Yale University Press, 1963 (1971)

> There is an updated edition available via amazon.com

• ‘Selling with color’, Faber Birren, McGraw-Hill Book Company, New York, 1945

> This book is NOT available via amazon.com, BUT his ‘Creative Color’ is.

• ‘Disruptive Pattern Material. An Encyclopedia of Camouflage’, Hardy Blechman,

Firefly Books, 2004, ISBN 1-55407-011-2

> Yes, the book is available via amazon.com

• ‘Peter Struycken’, Carel Blotkamp, NAi uitgevers, 2007, ISBN 978 90 5662 605 1

• ‘Color’, Betty Edwards, Penguin, 2004, ISBN 978-1-58542-219-7

> Yes, the book is available via amazon.com

• ‘Aristoteles over kleuren’, Ferweda & Strucken, Uitgeverij Damon, 2001

> This book is NOT available via amazon.com, BUT ‘Aristotle: Minor Works:

On Colours. On Things Heard. Physiognomics. On Plants. On Marvellous Things Heard.

Mechanical Problems. On Indivisible Lines. ... Gorgias (Loeb Classical Library

No. 307)’ is.

• ’Colour. Travels through the paintbox’, Victoria Finlay

> Yes, the book is available via amazon.com

• ‘Theory of Colours’, Johann Wolfgang von Goethe, MITT, paperback edition,

ISBN 0-262-57021-1

> Yes, the book is available via amazon.com

• ‘karl gerstner. Review of Seven Chapters of Constructive Pictures’, Eugen Gomringer,

Hatje Gantz, 2003, ISBN 3-7757-9151-5

> Yes, the book is available via amazon.com

• ‘The Elements of Color’, Johannes Itten, 1961 (1970)

> Yes, the book is available via amazon.com

• ‘Bart van der Leck’, Toos van Kooten, Museum Kröller Müller, 1994,

ISBN 90-73313-08-2

• ‘The new art – The new life. The collected writing of Piet Mondrian’,

Thames and Hudson, 1986

> Yes, the book is available via amazon.com

• ‘The Eye’s Mind. Collected writings 1965-1999’, Bridget Riley, Thames and Hudson,

1999, ISBN 0-500-28165-3

> Yes, the book is available via amazon.com

• ‘Frank Stella’, Robert Rosenblum, Penguin New Art 1, 1971, ISBN 0 14 070621 6

• ‘Dazling Painting. Kunst als camouflage’, Albert Roskam, Uitgeverij Van Spijk, 1987,

ISBN 90 7 1893 02 2

> Yes, the book is available via amazon.com

• ‘Frank Stella. Painting 1958 to 1965’, Lawrence Rubin, Stewart Tabori & Chang, 1986,

ISBN 0-941434-92-3

• ‘Peter Struycken. Trooping the colour’, —, Gorinchem, 2002, ISBN 90-804257-4-5

• ‘Orde en harmonie in het rijk der kleuren (an introduction to the color theory by

Ostwald)’, Tales & Zoon, Apeldoorn, 1927

on the web

– ‘As humans, our color vision influences everything from our art and poetry to the

colors’ via http://www.webexhibits.org/causesofcolor/1.html

– ‘Below are specifications for recommended color palettes’ via

http://www.stanford.edu/group/identity/ug_color.html

– ‘Colors are of philosophical interest for two kinds of reason’ via

http://plato.stanford.edu/entries/color/

– ‘Color can only exist when three components are present’ via

http://www.cambridgeincolour.com/tutorials/color-perception.htm

– ‘Color Perception Is Not In The Eye Of The Beholder: It's In The Brain’ via

http://www.sciencedaily.com/releases/2005/10/051026082313.htm

– ‘Color theory encompasses a multitude of definitions, concepts and design applications’

via http://www.colormatters.com/colortheory.html

– ‘Friedrich Wilhelm Ostwald was a Baltic German chemist’ via

http://en.wikipedia.org/wiki/Wilhelm_Ostwald

– ‘International Klein Blue (IKB) is a deep blue hue first mixed by the French artist Yves

Klein’ via http://en.wikipedia.org/wiki/International_Klein_Blue

Please also look at http://www.international-klein-blue.com/ and ‘Blue Women Art -

Yves Klein (1962)’ via http://www.youtube.com/watch?v=x0mYZbYdIpU

– ‘Introduction. At the moment, this site are best viewed with Internet Explorer 4.0.’

via http://www.cs.brown.edu/courses/cs092/VA10/HTML/start.html

– ‘Johannes Itten (11 November 1888 – 27 May 1967) was a Swiss expressionist

painter, designer, teacher, writer and theorist associated with the Bauhaus’ via

http://en.wikipedia.org/wiki/Johannes_Itten

Please also look at http://www.bauhaus.de/

– ‘Josef Albers (March 19, 1888 – March 25, 1976[1]) was a German-born American

artist and educator’ via http://en.wikipedia.org/wiki/Josef_Albers

– ‘Peter Struycken in the Groninger Musuem’ via

http://www.youtube.com/watch?v=1I6kKk_Ua30

Please also look at Skrjabin's Vision. Computer art combining music and visual arts.

Nine stills images from a dynamic colourspace by visual artist Peter Struycken based

upon the symphony Prometheus, poem of fire (1911) opus 60, by Alexander Skrjabin

1872-1915) via http://www.xs4all.nl/~kalden/stru/stru-skrjabin-E.html

– ‘Richard Paul Lohse was born in Zürich in 1902’ via http://www.lohse.ch/bio_e.html

– ‘The Department of Colour Science, founded in 1878, is an international centre of

excellence [. . . ]. It is unique in the UK’ via http://www.colour.leeds.ac.uk/

– ‘Theory of Colours is a book by Johann Wolfgang von Goethe published in 1810’ via

http://en.wikipedia.org/wiki/Theory_of_Colours

– ‘Where to Study Color (updated July, 2008)’ via

http://www.colormatters.com/des_studycolor.html

– ‘Why study color theory?’ via http://www.worqx.com/color/

papers

• D.H. Brainard, ‘Color Vision’

• Kate Bukoski, ‘Implications: seeing color, typography and color’

• Richard L. Gregory, ‘Images of mind in brain’

• C.L. Hardin & Luisa Maffi ‘Color categories in thought and language’

• John Harris, ‘How does visual memory work?’

• Paul Kay & Luisa Maffi ‘Color Appearance and teh Emergence and Evalution of Basic

Color Lexicons’

• Edwin H. Land, ‘The Retinex Theory of Color Vision’

• Cahrles Poynton ‘Frequently Asked Questions about Color’

• Dale Purves, ‘The Visual System and the Brian’

• Vilayanur S. Ramachandran & Edward M. Hubbard, ‘Hearing Colors, Tasting Shapes’

• Tech Museum, ‘All Those Seeing Color, Say Eye! Student Sheet’

• Wal Ting Slok & o, ‘Language regions of brain are operative in color perception’

• J. van de Weijer & Th. Gevers, ‘Color Constancy based on the Grey-Edge Hypothesis’