Multichannel vision system for estimating surface and illumination functions

Shoji Tominaga Vol. 13, No. 11 /November 1996 /J. Opt. Soc. Am. A 2163

Multichannel vision system for estimating surfaceand illumination functions

Shoji Tominaga

Osaka Electro-Communication University, Neyagawa, Osaka 572, Japan

Received December 13, 1995; revised manuscript received July 2, 1996; accepted July 8, 1996

This paper describes a set of experimental measurements and theoretical calculations designed to recover boththe surface-spectral reflectance function and the illuminant spectral-power distribution from the image data.A multichannel vision system comprising six color channels was created with the use of a monochrome CCDcamera and color filters. The spectral sensitivity of each color channel is calibrated, and the dynamic range ofthe camera is extended for sensing a wide range of intensity levels. Three algorithms and the correspondingresults are introduced. First, a method of choosing the appropriate dimension of the linear model dimensionsis introduced. Second, the illuminant parameters are estimated from the sensor measurements made at mul-tiple points within separate objects. Third, the sensor responses are corrected for highlight and shadingvariations. The body reflectance parameters, unique to each surface, are recovered from these corrected val-ues. Experimental results with a small number of test surfaces and a simple illumination geometry demon-strate that the illuminant spectrum and the surface-spectral reflectance functions can be recovered to withintypical deviations of 1% and 4%, respectively. © 1996 Optical Society of America.

1. INTRODUCTION

An experimental apparatus and a set of calculations forestimating both the illuminant spectral-power distribu-tion and the body reflectance function of surfaces insimple images are described. The apparatus consists of avision system with six spectral channels whose wave-length sensitivities cover the visible spectrum. The spec-tral channels were created by combining a monochromeCCD camera with (1) six different color filters for sam-pling of the spectrum and (2) a specially designed shutterfor extension of the dynamic range of the camera sensors.The calculations for estimating the wavelength functionsare based on approximations using low-dimensional lin-ear models (see, e.g., p. 296 of Wandell1; also, Refs. 2–7).There are three problems that we must solve in order

to use the camera data to estimate the surface and illu-minant information. First, we must decide on the dimen-sion of the linear models that we use to approximate thespectral functions. Second, we must adopt a procedurefor estimating the illumination. Third, we must identifyan algorithm for separating the body reflectance informa-tion from the highlight and shading information in thecamera data. This paper will describe algorithms foreach of these problems and then the experimental results.For the simple images studied here it is found that by us-ing this sequence of algorithms on the data and a six-channel system, one achieves an estimation accuracy ofmore than 99% for the illuminant and 96% for surface re-flectance.

2. EXPERIMENTAL APPARATUSA. Camera SystemFigure 1 shows the camera system for multichannel im-aging. The camera system is composed of a monochrome

0740-3232/96/1102163-11$10.00

CCD camera (Sony model TK-66), a standard photo-graphic lens (Nikon, focal length 50 mm, 35 mm), and sev-eral filters. The original CCD camera has been modifiedso that the shutter speed can be easily changed from theoutside. Two kinds of optical filter are used: a coloredglass filter for suppressing infrared light and some Kodakgelatin filters for selecting wavelength bands. The glassfilter is inserted between the lens and the camera, andthe gelatin filters are attached to the front of the lens.The linearity of the camera response was examined first.Uniform color patches displayed on a calibrated CRTmonitor were measured with both the camera and a radi-ometer. By a comparison of the camera outputs with theluminances of the incident light the camera was deter-mined to have a good linearity (g 5 1).

B. Measurement of the Spectral-Sensitivity FunctionKnowledge of the spectral-sensitivity function for eachsensor of the multichannel vision system is needed in es-timating the illuminant and surface-spectral reflectancefunctions. In particular, the spectral sensitivity of themonochrome CCD camera is important for predicting thewhole spectral sensitivity. Figure 2 shows the setup formeasuring the spectral sensitivity of the CCD camerawithout lenses. A set of 31 interference filters is used toconvert the continuous spectrum of a projector lamp intoa set of monochromes in 31 equally spaced wavelengthpoints throughout the visible region of 400–700 nm.Each interference filter has a limited transmittance. Themaximum transmittance ranges from 0.35 to 0.5, and thefull width at half-maximum of the transmittance functionis from 9 to 20 nm, which provides the amount of incidentlight adequate for the CCD sensor response. The mono-chromatic light is guided to diffusers, and the diffusedtransmission light is measured with both the camera andthe radiometer. The spectral-sensitivity function is then

© 1996 Optical Society of America

2164 J. Opt. Soc. Am. A/Vol. 13, No. 11 /November 1996 Shoji Tominaga

determined as the ratio of the camera output to the mea-sured radiance at each wavelength point. Figure 3shows the measured spectral-sensitivity function for themonochrome CCD camera.The present multichannel vision system has six differ-

ent color sensors. The number of sensors was motivatedfrom the model dimensions required for representation ofthe illuminant and reflectance spectral functions in afinite-dimensional linear model. By summarizing the re-sults previously reported,4–6 one can point out thatsurface-spectral reflectances of natural objects and artifi-cial ones can be represented with the use of five to sevenbasis functions. Along this line, Wratten filters are at-tached to the camera to separate the visible wavelengthinto six color bands. The set of filters is selected empiri-cally so that the color bands have sufficient transmissionlight for imaging, and the spectral-sensitivity functionsare different from each other. Figure 4 shows the wholespectral-sensitivity functions for the six sensors.

C. Dynamic Range ExtensionThe monochrome CCD camera that was used is sampledto 8 bits of intensity. While this is adequate for manypurposes, in experiments involving highlight estimationit is preferable to acquire more information about thehighlight data. I used a shutter to extend the dynamicrange of the CCD camera to sense the wide intensityvariation of object surfaces from matte parts to highlightparts. Pictures of the same scene are taken with differ-ent shutter speeds, and the multiple images are combinedinto a single image.

Fig. 1. Camera system for multichannel imaging.

Fig. 2. Setup for measuring the spectral sensitivity of the mono-chrome CCD camera.

For example, consider three pictures taken at shutterspeeds of 1/125, 1/250, and 1/500, respectively, i.e., threeexposure times of 8, 4, and 2 msec, in 1/2 steps. Let1/125 s be the standard shutter speed. Moreover let pijand p ij

(k) indicate the (i, j) pixel values in the combinedimage and in the kth-millisecond image, respectively.First, if pij

(8) , pmax (5255), then we accept p ij(8) as pij

5 22 3 pij(8) . Second, if pij

(8) > pmax and pij(4) , pmax ,

then we discard p ij(8) and accept p ij

(4) as pij 5 21 3 pij(4) .

Third, if pij(4) > pmax and pij

(2) , pmax , then pij 5 pij(2) .

This procedure is usually accurate, because quantizationerrors and noise are not magnified.

3. DEFINITIONSThe color signal C(x, l) from an object surface is a func-tion of the wavelength l, ranging over a visible wave-length, and the location parameter x, including variousgeometric parameters under a fixed imaging geometry.The dichromatic reflection model suggests that light re-

Fig. 3. Spectral-sensitivity function measured for the mono-chrome CCD camera.

Fig. 4. Spectral-sensitivity functions for the six sensors. R,red; B, blue; G, green; Y, yellow.


flected from the surface of an inhomogeneous dielectricobject is composed of two additive components, the bodyreflection and the interface reflection.8–12 The color sig-nal C(x, l) is then expressed in the form

C~x, l! 5 a~x !S~l!E~l! 1 b~x !E~l!, (1)

where S(l) is the surface-spectral reflectance function ofan object and E(l) is the spectral-power distribution ofthe illumination. The first term on the right-hand side ofEq. (1) represents the body reflection, where a (x) is itsshading factor, while the second term represents thespecular reflection, where b (x) is the scale factor.We use linear finite-dimensional models to describe the

spectral functions. Specifically, we suppose that the illu-minant E(l) can be expressed in a linear combination ofm basis functions as

E~l! 5 (i51

m

« iEi~l!, (2)

where $Ei(l), i 5 1, 2, . . ., m% is a statistically deter-mined set of basis functions for the illuminant and $«i% isa set of scalar weights. We also suppose that the spectralreflectance function S(l) can be expressed in the samefashion with n reflectance basis functions as

S~l! 5 (j51

n

s jSj~l!, (3)

where $Sj(l), j 5 1, 2, . . ., n% is a set of basis functionsfor reflectance and $sj% is the weight set. The number ofbasis functions, m or n, defines the model dimension forilluminants or surfaces, respectively.There are six sensor outputs at each spatial location x.

The sensor outputs, rk(x), the sensor spectral-sensitivityfunctions, Rk(l), and the color signal are related by theequation

rk~x ! 5 E C~x, l!Rk~l!dl, k 5 1, 2, . . ., 6. (4)

The spectral-sensitivity functions of the k sensors in thepresent system are shown in Fig. 4.Finally, it is convenient to summarize the imaging re-

lationships between the sensor responses and the illumi-nant and surface functions in a matrix form. First, de-fine six-dimensional vectors hi (i 5 1, 2, . . ., m)representing the sensor responses for the illuminant ba-sis as

h1 5 S E E1~l!R1~l!dl

•

•

•

E E1~l!R6~l!dlD , . . .,

hm 5 S E Em~l!R1~l!dl

•

•

•

E Em~l!R6~l!dlD . (5)

These are summarized in a 6 3 m matrix H:

H 5 ~h1 , h2 , . . ., hm!. (6)

Next, define a 6 3 n matrix L« representing the sensorresponse for the color basis as

L« 5 3 (i51

m

«1E Ei~l!S1~l!R1~l!dl • • • (i51

m

«1E Ei~l!Sn~l!R1~l!dl

•

•

•

•

•

•

(i51

m

« iE Ei~l!S1~l!R6~l!dl • • • (i51

m

« iE Ei~l!Sn~l!R6~l!dl4 . (7)

Note that hi (i 5 1, 2, . . ., m) in Eqs. (5) and the integralsin Eq. (7) are known ahead of time with the spectral-sensitivity functions of sensors and the basis functions forthe illuminant and reflectance function.Substituting the finite-dimensional model expressions

for E(l) and S(l) into Eq. (4) and using the above matrixnotations permit us to express the general imaging rela-tionship between the sensor outputs and the scene pa-rameters by means of the vector equation

r~x ! 5 a~x !L«s 1 b~x !H«, (8)

where r(x) is a column vector formed from six sensor re-sponses rk(x) (k 5 1, 2, . . ., 6). The vectors s and «are of dimensions n and m, respectively, and they repre-sent the weight vectors for surface reflectance and the il-luminant.

4. ALGORITHMSGiven the formulation of the problem in terms of linearmodels, the estimation problem becomes one of inferringthe illuminant vector « and the reflectance vector s fromthe sensor output vectors r(x) of the image. Three stepsare involved in the estimation problem.First, we define an algorithm to select the finite-

dimensional linear model. When we use six sensors, themodel dimensions should be bounded to be 6 or fewer.But this is only an upper limit. The presence of sensornoise and quantization error may make it worse to usemodels with the full dimensionality. In particular, thecontribution to the sensor response from each of the lin-


ear model terms decreases generally as one adds more ba-sis functions. In this case the cost of the variance of thecoefficient estimate, caused by noise and quantization er-rors, may exceed the benefit from the additional precisionof the high-dimensional linear model. Because the esti-mation process used here begins by recovering the illumi-nation, the first algorithm introduced below is a methodof selecting the linear model dimension for the illuminantfunctions.Second, we define an algorithm to estimate the illumi-

nant. In previous work Tominaga and Wandell10 (seealso Lee11 and D’Zmura and Lennie13) described a methodof estimating the illumination based on sensor responsescollected from two surfaces containing specular high-lights. Here we extend the algorithm in Ref. 10 so that itapplies to measurements from multiple surfaces.Third, we define an algorithm to estimate the body re-

flectance function of the surfaces in the image. To esti-mate the body reflectance function, we must remove theeffects of specularity and shading and then segment theimage into uniform surface areas.

A. Linear Model Dimension SelectionThe appropriate linear model dimension will depend onthe properties of the representative illuminants and onthe properties of the measurement instrument. The abil-ity of the vision system to estimate the illuminant andsurface functions will be limited by factors such as spec-tral responsivity of the camera sensors, noise, and quan-tization. In the development of a measurement system itis useful to have a method that includes all of these fac-tors when one decides on the number of linear model di-mensions.Suppose that we model the sensor outputs for a white

reflectance standard as the noisy measurements

rw 5 (i51

m

« ihi 1 h, (9)

where h is a noise vector, containing measurement errorsand model-fitting errors. The location parameter x is ne-glected by assuming the uniformity of the white surfaceand illumination. From a statistical viewpoint Eq. (9) isregarded as the equation of a linear multiple regressionmodel. We will solve the problem of deciding on the di-mension of the linear model, m, by using a regressionanalysis.14

Let us consider a nested pair of regression models inwhich the smaller model has one fewer linear model di-mension:

Null hypothesis: rw 5 (i51

m21

« ihi 1 h,

Alternative hypothesis: rw 5 (i51

m21

« ihi 1 «mhm 1 h.

The smaller model can be obtained from the larger by set-ting a weighting coefficient in the larger model equal tozero. We will compare the smaller model (null hypoth-esis) with the larger model (alternative hypothesis). Ifthe alternative model is significantly better than the nullmodel, then the additional dimension, represented by

«mhm , improves the linear model and should be used.Otherwise, we need not add any further dimensions to thelinear model.To decide between these two models, we use the statis-

tical index F. The F value for six sensors is given by theequation

F 5«m

2

a ~m,m !@Jm /~6 2 m !#, (10)

where the scalar «m is the mth of the least-squared esti-mates «1 , «2 , . . ., «m , the scalar a (m,m) is the (m, m)element of the m 3 m inverse matrix (H1H)21, and Jm isthe residual

Jm 5 I rw 2 (i51

m

« ihiI 2. (11)

If the noise h is normally and independently distributedwith zero mean and a constant variance, then under thenull hypothesis, the F value will follow an F distributionwith degrees of freedom associated with the numeratorand the denominator of Eq. (10), 1 and 6 2 m. That is,we have F ; F(1, 6 2 m). Thus we can use the percent-age points of the F distribution to assign a significancelevel to the F value. The F test then gives evidenceagainst the null hypothesis and for the alternative hy-pothesis if the F value is large when compared with thepercentage points.The iterative procedure for deciding on the model di-

mension is summarized as follows:

1. Collect a set of sensor measurements from a knowntarget of the white surface.2. The basis functions Ei(l) are arranged in descend-

ing order of principal components.3. Suppose that the model currently includes m 2 1

terms of h1, h2 , . . ., hm21. Use the measurements tocompute the least-squared estimates «1 , «2, . . ., «m in alinear combination of the m terms h1 , h2, . . ., hm , andthe corresponding F value.4. If F . F* , where F* is the a 3 100% point of the

F(1, 6 2 m) distribution, we accept the large model in-stead of the small model and add the term hm to themodel. The choice of a 5 0.05 is usual.5. We continue adding one term until a stopping rule

is met. If F , F* , then hm is not added to the model.The iterative process is terminated, and the model is pre-dicted to have the dimension of m 2 1.

B. Illuminant Estimation AlgorithmThe dichromatic imaging model defined in Eq. (8) showsthat the sensor output r(x) for any spatial location x on anobject is expressed as a linear combination of the two re-flection component vectors L«s and H«. These two vec-tors span a two-dimensional subspace (plane) in a six-dimensional sensor space. This subspace is called thecolor–signal plane P.Suppose that the image sensors observe M objects illu-

minated with one light source and that the areas withhigh intensity are extracted from the image data. TheM


color–signal planes must intersect because H« is con-tained in all planes P(1), P(2), . . ., P(M) (see Fig. 5).Once the intersection is found, the illuminant vector « isrecovered by applying a matrix inversion H21 or apseudomatrix inversion H1 to H«. Tominaga andWandell10 presented an algorithm for finding the inter-section of two planes in the case of two objects. Here asolution method is presented for finding a reliable esti-mate of the common intersection vector in any case ofmore than two objects.Let M pairs of vectors [u1

(1) , u2(1)], [u1

(2) , u2(2)], . . .,

[u 1(M), u 2

(M)] be the orthonormal bases that define theplanes P(1), P(2), . . ., P(M), respectively. The basis vectorscan be obtained computationally from the singular-valuedecomposition of the observed data. When P(1),P(2), . . ., P(M) intersect at a common line, the intersectionline must lie in all M planes. An intersection betweentwo planes is represented in one equation, and so thecommon intersection among M planes must satisfy (M2

2 M)/2 equations in all combinations of any two planes.This requirement is formalized as

c1~i !u1

~i ! 1 c2~i !u2

~i ! 5 c1~ j !u1

~ j ! 1 c2~ j !u2

~ j ! for i Þ j.(12)

These equations can be grouped into a single set of homo-geneous linear equations:

F u1~1 !

0•

•

•

u1~1 !

u2~1 !

0•

•

•

u2~1 !

2u1~2 !

u1~2 !

•

•

•

0

2u2~2 !

u2~2 !

•

•

•

0

0

2u1~3 !

•

•

•

0

0

2u2~3 !

•

•

•

0

•••

•••

•••

•••

0

0•

•

•

2u1~M !

0

0•

•

•

2u2~M !

G S c1~1 !

c2~1 !

c1~2 !

c2~2 !

•

•

•

c1~M !

c2~M !

D 5 0. (13)

It can be shown that solving Eq. (13) is equivalent to find-ing the (2M)th vector of the right-hand singular matrixfrom the following 2M 3 2M symmetrical matrix, whoseentries are all scalars:

3M 2 1

0

2u1~1 !tu1

~2 !

2u1~1 !tu2

~2 !

•

•

•

2u1~1 !tu1

~M !

2u1~1 !tu2

~M !

0

M 2 1

2u2~1 !tu1

~2 !

2u2~1 !tu2

~2 !

•

•

•

2u2~1 !tu1

~M !

2u2~1 !tu2

~M !

2u1~1 !tu1

~2 !

2u2~1 !tu1

~2 !

M 2 1

0•

•

•

2u1~2 !tu1

~M !

2u1~2 !tu2

~M !

2u1~1 !tu2

~2 !

2u2~1 !tu2

~2 !

0

M 2 1•

•

•

2u2~2 !tu1

~M !

2u2~2 !tu2

~M !

•••

•••

•••

•••

•••

•••

•••

2u1~1 !tu1

~M !

2u2~1 !tu1

~M !

2u1~2 !tu1

~M !

2u2~2 !tu1

~M !

•

•

•

M 2 1

0

2u1~1 !tu2

~M !

2u2~1 !tu2

~M !

2u1~2 !tu2

~M !

2u2~2 !tu2

~M !

•

•

•

0

M 2 1

4 .
Once the coefficients c 1
(i) and c 2(i) are estimated from this

calculation, a reliable estimate of the intersection vector e

is the mean value

e 51M (

i51

M

@c1~i !u1

~i ! 1 c2~i !u2

~i !#. (14)

The illuminant vector can then be estimated from theequation

« 5 H1e. (15)

C. Image SegmentationFirst, the influence of illumination is reduced from thesensor outputs as

Fig. 5. Intersection of multiple color–signal planes.

y~x ! 5 r~x !/e, (16)

where e is an estimate of the illuminant vector H « and

the division is done elementwise. The above equationmeans a component-by-component division of the ith sen-


sor value at a point in the image by the ith sensor valuethat is due to a white reference under the same illumina-tion. This is analogous (in six sensor dimensions) to vonKries adaptation in vision models. With a reliable esti-mate e the above division makes y(x) 5 a(x)(L«s/H«)1 b(x)i, where i is a constant vector with unit length. Ifthe spectral-sensitivity functions of the sensors have nar-row wavelength bands, the vector y(x) is independent ofthe illumination.Next, we normalize this modified sensor output y(x) to

eliminate the constant interface reflection term and thegeometric factors a (x) and b (x). This normalizationprocess consists of two steps: First, define a deviationvector as y(x) 2 y(x), where y(x) denotes the averagewhose entries are a constant average value S i51

6 yi/6. Inthis step the interface reflection term is deleted. Second,normalize the deviation vector to a unit-length vector as

y~x ! 5y~x ! 2 y~x !

iy~x ! 2 y~x !i, (17)

where the notation i•i is defined as iyi2 5 yty. Here thegeometric factor to the deviation vectors is canceled out.The normalized sensor output y(x) is independent of

the vector H« and the scalars a (x) and b (x), so that thisvector depends heavily on only the body reflectance of anobject surface. This normalization property permits usto classify the sensor outputs measured from different lo-cations in a scene on the basis of surface materials. Inother words, we can segment the image into uniform sur-face areas.This image segmentation is done by means of cluster

detection in the six-dimensional space of the normalizedvector y(x). Dense clusters corresponding to uniformsurface areas are detected from the histogram analysis inthe space. This process is similar to the cluster detectionfor partitioning a color image into a set of uniform colorareas. The basic process is based on sequential clusterdetection in a three-dimensional color space.15 For ex-ample, Tominaga16 presented a detection algorithm by aniterative analysis of a one-dimensional histogram. Wecan extend such an algorithm to the cluster analysis inthe six-dimensional space. Conversely, if we extract thefirst three principal components of the distribution ofy(x), the problem of cluster detection in the y(x) spacemay be simplified to the cluster detection in the three-dimensional feature space defined by the principal compo-nents. In this case the previous algorithms for color im-age segmentation can be applied directly to the presentimage segmentation.

Fig. 6. Coordinate system of (ce ,c e').

D. Reflectance Estimation AlgorithmFor each area in the segmented image we estimate aunique reflectance function from the classified sensor out-puts as one surface. The sensor outputs are analyzed onthe color–signal plane. Because of the two dimensional-ity, the body reflection vector L«s can be expressed interms of two orthonormal vectors e and e' as

L«s 5 cee 1 ce'e', (18)

where e was obtained as the estimate ofH« in Subsection4.B and e' is a unit-length vector perpendicular to e onthe plane. The scalars ce and c e

' are the weighting coef-ficients. A permissible solution for the coefficients isgiven by applying an extended quarter-circle analysis.17,18

Figure 6 demonstrates the coordinate system of(ce , c e

'). Each dot indicates the sensor response to loca-tion x on a surface in this transformed-coordinate frame.The ce axis represents the weight of the interface reflec-tion, that is, the illumination, while the c e

' axis repre-sents the contribution that is due to the body reflection.Therefore a reliable estimate of the vector L«s is obtainedas the data point of sensor responses on the plane that isfarthest from the illuminant vector e. In Fig. 6 the mostreliable estimate is given at an intersection betweenboundaries b1 and b2 . The coordinate point (ce , c e

') isdescribed by the equations

c e 5 @max~cex' !#/@max~cex

' /cex!#, c e' 5 max~cex

' !.(19)

Finally, the estimate of the reflectance parameter s isgiven as

s 5 L«1 ~ c ee 1 c e

'e'!, (20)

where L«1 is the n 3 6 pseudoinverse of L« .

5. EXPERIMENTAL RESULTSTo derive the illuminant basis functions, I analyzed a setof spectra from CIE standard lights and several realsources. The data set consisted of the following sources:

1. CIE standard lights A, B, and C.2. CIE daylights D55, D65, and D75.3. Measured spectra from sunlight, a slide projector,

and a tungsten halogen lamp.

Figure 7 shows the first three basis functions computedfrom the singular-value decomposition of the set of ninespectra. Each basis curve was sampled at intervals of 5nm, and the illuminant basis [Ei(l)] was represented as61-dimensional vectors. The percent variance of the ba-sis vectors was 0.9696 for the first two components and0.9989 for the first three components.The reflectance basis functions were determined with

the use of a database of surface-spectral reflectances pro-vided by Eastman Kodak Company. This database con-sisted of 354 measured reflectance spectra of differentmaterials collected from Munsell chips, paint chips, andnatural objects. Vrhel et al.6 analyzed, throughprincipal-component analysis, this set of reflectance spec-


tra in detail and suggested that fewer than seven basisvectors might be sufficient for modeling of the reflectancespectra. I computed the reflectance basis $Sj(l)% by thedirect singular-value decomposition of the set of mea-sured reflectance spectra. Figure 8 shows the first fivebasis functions. The percent variance was 0.9964 for thefirst four components and 0.9980 for the first five compo-nents.

A. Linear Model Dimension EstimationI have estimated the dimension of the illuminant modelfor the halogen lamp used in the present vision system.The white standard plate was measured with the cameraunder uniform lighting. The sensor outputs for 11 3 11pixels from a small area of the surface were examined onthe computations for the statistical F test in Subsection3.A. Figure 9 shows the histogram of the predicted di-mensions from the image data. In the figure the fre-quency indicates the total number of dimensions assigned

Fig. 7. First three basis functions for the set of nine illuminantspectra.

Fig. 8. First five basis functions for the database of reflectancespectra.

to m over 121 measurements. From the above distribu-tion of dimensions we can estimate the model dimensionas m 5 3.The validity of this dimension estimation is confirmed

experimentally. The estimation accuracy of the illumi-nant depends clearly on the model dimension used. Nowlet « 1

(m) , « 2(m) , . . ., « m

(m) be the estimated weights for thebasis functions under the assumption of the model dimen-sion m. The performance of the illuminant estimation isthen evaluated by the squared error

Je~m ! 5 IE~l! 2 (i51

m

« i~m !Ei~l!I 2. (21)

I have examined this performance by using the same im-age data and spectroradiometer measurements as the ex-periment described below in Subsection 5.B.1, which usesplastic targets under a halogen lamp. In Fig. 10 thevariation of the error Je(m) is shown as a function of m.

Fig. 9. Histogram of the predicted dimension for the illuminantmodel of a halogen lamp from the image data.

Fig. 10. Variations of the estimation error Je(m) and the ap-proximation error Ja(m) as a function of the dimensionm for theilluminant spectrum of a halogen lamp. The computations ofJe(m) were done with the use of the experimental data in Sub-section 5.B.1.


It should be noted that increasing the number m does notalways lead to decreasing the estimation error. The es-timation accuracy improves at first asm increases, but af-ter m 5 3 it worsens. In the figure the variation of theestimation error is compared with the variation of the ap-proximation error, defined as

Ja~m ! 5 IE~l! 2 (i51

m

« iEi~l!I 2, (22)

where the weights «1 , «2 , . . ., «m are calculated directlyfrom the orthogonal expansion of E(l) without the use ofthe image data. Note that the error Ja(m), marked withplus signs in Fig. 10, decreases monotonically with m, incontrast with the error Je(m).Next, the relationship between the model dimensions

and the estimation accuracy is examined for surface-spectral reflectance in the same experiment with the useof three surfaces of plastic. The estimated reflectanceparameters depends on both the numbers m and n ass1(m,n), s2

(m,n), . . ., sm(m,n). The estimation error of the

surface reflectance function is represented as

Js~m, n ! 5 IS~l! 2 (i51

n

s i~m,n !Si~l!I 2. (23)

Figure 11 shows the average value of Js(m, n) over threesurfaces as a function of m and n on a logarithmic scale.All the estimates in the range of 2 < m, n < 6 were com-puted in the same way as that in Subsection 5.B.1. Noteagain that the performance of reflectance estimation de-pends greatly on the model dimensions, while the math-ematical approximation of spectral reflectance is im-proved with only the reflectance dimension n. The aboveresult suggests that the most appropriate dimensions forminimizing the error in reflectance estimation are givenat approximately m 5 3 and n 5 5.

B. ResultsTo test the multichannel vision system and the proposedestimation algorithm, I conducted two experiments, usingdifferent kinds of material. One object is three sheets ofplastic, and the other is three sheets of paper. I put ob-jects on a cylinder to make the plastic cylinders and the

Fig. 11. Average value of the estimation error Js(m, n) overthree surfaces of plastic as a function of m and n.

paper cylinders with three colors. Two light sourceswere used. The lamps are placed at the same height asthe objects so that the side of the cylinder on a table isilluminated. The camera is also placed at the sameheight, and the viewing direction is almost coincidentwith the illumination.

1. Experiments with Plastic TargetsFigure 12 shows the scene of red, green, and yellow plas-tic cylinders illuminated with a halogen lamp. Specularhighlights appear on the side of the cylinders. Two im-ages of this scene were taken with the multichannel cam-era at the two shutter speeds of 1/125 and 1/250 s, andthen the two images were combined by a simple scaling.Figure 13 shows the color histogram of the combined im-age. In the figure a three-dimensional subspace spannedby the first three principal-component vectors is used forconvenient three-dimensional displaying of the six-

Fig. 12. Scene of red, green, and yellow plastic cylinders illumi-nated with a halogen lamp.

Fig. 13. Color histogram of the combined image of the plasticcylinders in a three-dimensional subspace.


dimensional image data. Three long, straight clusterscorrespond to the areas of highlights on three color ob-jects. I extracted three areas, including highlights fromthe measured image, and obtained two basis vectors forthe respective areas. The intersection e of three color–signal planes P(1), P(2), and P(3) was computed from thethree sets of basis vectors. Figure 14 shows the estima-tion results of the illuminant spectral-power distribution,where the illuminant estimate from the image data is rep-resented as the curve marked with squares. We comparethe estimate with a direct measurement of the illuminantspectrum of the halogen lamp used. A standard whitesample, placed at the same position as that of the objects,was measured with a spectroradiometer. The spectraldistribution from this measurement is represented byplus signs in the figure. An almost perfect coincidence isseen between the two curves.Next, I computed the normalized deviation vectors y(x)

of the sensor outputs to reduce the influence of illumina-tion from the measured image. Figure 15 shows the his-togram of the normalized sensor outputs in a three-dimensional subspace spanned by the first three principal

Fig. 14. Estimation results of the illuminant spectral-power dis-tribution of a halogen lamp. Squares represent the estimatefrom the image data, and plus signs represent the direct mea-surement by a spectro-radiometer.

Fig. 15. Histogram of the normalized sensor outputs for the im-age of the plastic cylinders in a three-dimensional subspace.

components. The three clusters in the figure correspondto the observations from the three color cylinders. It canbe seen that the original color distributions with largevariations in Fig. 13 are reduced to compact clusters inthe coordinates of the normalized sensor outputs. Theseclusters are well separated from each other for classifica-tion. The original image was segmented based on thisclassification, and the regions of three cylinder surfaceswere extracted.A unique reflectance function for each surface was de-

termined from the extracted image data. Figure 16shows the coordinates of the weighting coefficients(ce , c e

') on e and e'. The number of the original coordi-nate points was reduced by thinning the clusters. In thefigure the reduced number of points are plotted with plussigns on a common scale. The estimates (ce , c e

') of theweighting coefficients were calculated, based on theseweight distributions of the observed data. The estimatedcoordinate points are plotted with squares in the samefigure. Finally, the spectral curves of the estimated re-

Fig. 16. Coordinates of the weighting coefficients (ce , c e') for

three color regions.

Fig. 17. Estimation results of the surface-spectral reflectancefunctions for three surfaces of the plastic cylinders. Diamondsrepresent the estimated spectral reflectance curves from the im-age data, plus signs represent the measured spectral reflectancecurves, and squares represent the five-dimensional model of thereflectance curves.


flectance functions are depicted for the respective sur-faces in Fig. 17, where diamonds represent the estimates.To confirm the reliability of these results, we compare thepresent estimates with the measurement results, usingthe spectrometer and the standard white surface. In Fig.17 the curves with plus signs represent the measurementresults, and the curves with squares represent the linearfinite-dimensional model of the measured spectral reflec-tance curves with the use of the five basis functions of Fig.8. It is remarked that the two curves with diamonds andsquares are close for all the color surfaces. The averagediscrepancy is approximately 0.023. We can concludethat the estimation results are reliable, as the presentsystem uses the five-dimensional linear model for reflec-tance description.

2. Experiments with Paper TargetsIt was possible to use the same method for the paper cyl-inders as that in Subsection 5.B.1. Figure 18 shows thescene of red, green, and yellow paper cylinders illumi-nated with a slide projector. Little specular reflectionappears on the side of the cylinders. An image was takenof the scene with the camera at a single shutter speed.Figure 19 shows the color histogram of the image in a

Fig. 18. Scene of red, green, and yellow paper cylinders illumi-nated with a slide projector.

Fig. 19. Color histogram of the image of the paper cylinder in athree-dimensional subspace.

three-dimensional subspace. Most pixels of the threeclusters come from matte surfaces of the papers. Threeareas with high luminous intensities were extracted fromthe image, and an intersection of the color–signal planeswas computed. Figure 20 shows the estimation results ofthe illuminant spectral-power distribution of a slide-projector lamp. In the figure the three curves markedwith diamonds, plus signs, and squares represent, respec-tively, the estimate, the direct measurement, and thethree-dimensional model of the measurement with theuse of the three basis functions of Fig. 7. A comparison ofthe three curves suggests that the surfaces of the papershave the property of the standard dichromaticreflection,19 and the illumination spectrum can be esti-mated from these papers. The average discrepancy be-tween the estimate and the three-dimensional model isapproximately 0.0048.

Fig. 20. Estimation results of the illuminant spectral-power dis-tribution of a slide projector. Diamonds, plus signs, and squaresrepresent, respectively, the estimate, the direct measurement,and the three-dimensional model of the measurement.

Fig. 21. Estimation results of the surface-spectral reflectancefunctions for three surfaces of the paper cylinders. Squares andplus signs represent, respectively, the estimate and the five-dimensional model of the measured reflectance curves.


Next, the surface-spectral reflectance functions for thethree papers were estimated from the segmented imagedata in the same manner as in the experiment of Subsec-tion 5.B.1. Figure 21 shows the reflectance estimationresults, where squares represent the estimates from theimage data and plus signs represent the five-dimensionalmodels of the measured spectral reflectances. A com-parison between the two curves for each surface indicatesthe reliability of the reflectance estimation for papers.The average discrepancy is approximately 0.032.

6. CONCLUSIONThis paper has described a set of experimental measure-ments and theoretical calculations designed to recoverboth the surface-spectral reflectance function and the il-luminant spectral-power distribution from the imagedata. A multichannel vision system, which consisted ofsix sensors over the visible wavelength, was realized bycombining a monochrome CCD camera and six sets ofcolor filters. The spectral sensitivity of each sensor wascalibrated, and the dynamic range was extended for sens-ing a wide range of intensity levels including highlights.Three algorithms and the corresponding results were

introduced so that one can use the camera data to esti-mate the surface and illuminant information. First, amethod was introduced of choosing the appropriate di-mension of the linear models used for approximation ofthe spectral functions. Note that increasing the modeldimensions does not always lead to improving the estima-tion accuracy. The appropriate dimensions in thepresent system are approximately 3 for the illuminantand 5 for reflectance. Second, the illuminant parameterswere estimated from the sensor measurements made atmultiple points within separate objects. Third, the sen-sor responses were corrected for highlight and shadingvariations, and the body reflectance parameters, uniqueto each surface, were recovered from these corrected val-ues. Experimental results for simple images demon-strated that by using the multichannel vision system andthe above sequence of proposed algorithms, one achievedan estimation accuracy of more than 99% for the illumi-nant and 96% for surface reflectance.

ACKNOWLEDGMENTThe author thanks Brian Wandell of Stanford Universityfor his helpful discussions and comments on this manu-script.

The author can be reached by telephone at 181-720-20-4562, by fax at 181-720-24-0014, and by e-mail:[email protected].

REFERENCES1. B. A. Wandell, Foundations of Vision (Sinauer Associates,

Sunderland, Mass., 1995).2. D. B. Judd, D. L. MacAdam, and G. W. Wyszecki, ‘‘Spectral

distribution of typical daylight as a function of correlatedcolor temperature,’’ J. Opt. Soc. Am. 54, 1031–1040 (1964).

3. J. Cohen, ‘‘Dependency of the spectral reflectance curves ofthe Munsell color chips,’’ Psychon. Sci. 1, 369–370 (1964).

4. L. T. Maloney, ‘‘Evaluation of linear models of surface spec-tral reflectance with small numbers of parameters,’’ J. Opt.Soc. Am. A 10, 1673–1683 (1986).

5. J. P. S. Parkkine, J. Hallikainen, and T. Jaaskelainen,‘‘Characteristic spectra of Munsell colors,’’ J. Opt. Soc. Am.A 6, 318–322 (1989).

6. M. J. Vrhel, R. Gershon, and L. S. Iwan, ‘‘Measurement andanalysis of object reflectance spectra,’’ Color Res. Appl. 19,4–9 (1994).

7. B. Funt, ‘‘Linear model and computational color constancy,’’in Proceedings of IS&T/SID Color Imaging Conference (In-formation Science and Technology, Springfield, Va., 1995),pp. 26–29.

8. S. A. Shafer, ‘‘Using color to separate reflection compo-nents,’’ Color Res. Appl. 10, 210–218 (1985).

9. G. J. Klinker, S. A. Shafer, and T. Kanade, ‘‘The measure-ment of highlights in color images,’’ Int. J. Comput. Vis. 2,7–32 (1988).

10. S. Tominaga and B. A. Wandell, ‘‘The standard surface re-flectance model and illuminant estimation,’’ J. Opt. Soc.Am. A 6, 576–584 (1989).

11. H. C. Lee, ‘‘Method for computing the scene-illuminantchromaticity from specular highlights,’’ J. Opt. Soc. Am. A3, 1694–1699 (1986).

12. G. Healey, ‘‘Estimating spectral reflectance using high-lights,’’ Image Vision Comput. 9, 333–337 (1991).

13. M. D’Zmura and P. Lennie, ‘‘Mechanisms of color con-stancy,’’ J. Opt. Soc. Am. A 3, 1662–1682 (1986).

14. S. Weisberg, Applied Linear Regression (Wiley, New York,1980).

15. M. D. Levine, Vision in Man and Machine (McGraw-Hill,New York, 1985).

16. S. Tominaga, ‘‘Color classification of natural color images,’’Color Res. Appl. 17, 230–239 (1992).

17. S. Tominaga and B. Wandell, ‘‘Component estimation ofsurface spectral reflectance,’’ J. Opt. Soc. Am. A 7, 312–317(1990).

18. S. Tominaga, ‘‘Surface identification using the dichromaticreflection model,’’ IEEE Trans. Pattern Mach. Intell. 13,658–670 (1991).

19. S. Tominaga, ‘‘Dichromatic reflection models for a variety ofmaterials,’’ Color Res. Appl. 19, 277–285 (1994).

Multichannel vision system for estimating surface and illumination functions

Documents