Color and Brightness Constancy Jim Rehg CS 4495/7495 Computer Vision Lecture 25 & 26 Wed Oct 18, 2002.

Color and Brightness ConstancyColor and Brightness Constancy

Jim RehgJim Rehg

CS 4495/7495 Computer VisionCS 4495/7495 Computer Vision

Lecture 25 & 26Lecture 25 & 26

Wed Oct 18, 2002Wed Oct 18, 2002

2 J. M. Rehg © 2002

OutlineOutline

Human color inferenceHuman color inference Land’s RetinexLand’s Retinex Dichromatic reflectance modelDichromatic reflectance model Finite dimensional linear modelsFinite dimensional linear models Color constancy algorithmColor constancy algorithm

3 J. M. Rehg © 2002

Human Color ConstancyHuman Color Constancy

Distinguish betweenDistinguish between Color constancy, which refers to hue and saturationColor constancy, which refers to hue and saturation Lightness constancy, which refers to gray-level.Lightness constancy, which refers to gray-level.

Humans can perceiveHumans can perceive Color a surface would have under white light Color a surface would have under white light

(surface color)(surface color) Color of the reflected light (limited ability to separate Color of the reflected light (limited ability to separate

surface color from measured color)surface color from measured color) Color of illuminant (even more limited)Color of illuminant (even more limited)

4 J. M. Rehg © 2002

Spatial Arrangement and Color Spatial Arrangement and Color PerceptionPerception

5 J. M. Rehg © 2002

Spatial Arrangement and Color Spatial Arrangement and Color PerceptionPerception

6 J. M. Rehg © 2002

Spatial Arrangement and Color Spatial Arrangement and Color PerceptionPerception

7 J. M. Rehg © 2002

Land’s Mondrian ExperimentsLand’s Mondrian Experiments

The (by-now) familiar phenomena: Squares of The (by-now) familiar phenomena: Squares of color with the same color radiance yield very color with the same color radiance yield very different color perceptionsdifferent color perceptions

Photometer: 1.0, 0.3, 0.3 Photometer: 1.0, 0.3, 0.3

Audience: “Red” Audience: “Blue”White light Colored light

8 J. M. Rehg © 2002

Basic Model for Lightness Basic Model for Lightness ConstancyConstancy

Modeling assumptions for cameraModeling assumptions for camera Planar frontal scenePlanar frontal scene Lambertian reflectanceLambertian reflectance Linear camera response Linear camera response

Camera model:Camera model: Modeling assumptions for sceneModeling assumptions for scene

Albedo is piecewise constantAlbedo is piecewise constant– Exception: ripening fruitException: ripening fruit

Illumination is slowly-varyingIllumination is slowly-varying– Exception: shadow boundariesException: shadow boundaries

)()()( xpxIkxC c

9 J. M. Rehg © 2002

Algorithm ComponentsAlgorithm Components

The goal is to determine what the surfaces in the The goal is to determine what the surfaces in the image would look like under white light.image would look like under white light.

A process that compares the brightness of patchs A process that compares the brightness of patchs across their common boundaries and computes across their common boundaries and computes relative brightness.relative brightness.

A process that establishes an absolute reference A process that establishes an absolute reference for lightness (e.g. brightest point is “white”)for lightness (e.g. brightest point is “white”)

10 J. M. Rehg © 2002

1-D Lightness “Retinex”1-D Lightness “Retinex”

Threshold gradient image to find surface (patch) boundaries

11 J. M. Rehg © 2002

1-D Lightness “Retinex”1-D Lightness “Retinex”

Integration to recover surface lightness (unknown constant)

12 J. M. Rehg © 2002

Extension to 2-DExtension to 2-D

Spatial issuesSpatial issues Integration becomes much harderIntegration becomes much harder

– Integrate along many sample paths (random walk)Integrate along many sample paths (random walk)– Loopy propagationLoopy propagation

Recover of absolute lightness/color referenceRecover of absolute lightness/color reference Brightest patch is whiteBrightest patch is white Average reflectance across scene is knownAverage reflectance across scene is known Gamut is knownGamut is known Specularities can be detectedSpecularities can be detected Known reference (color chart, skin color, etc.)Known reference (color chart, skin color, etc.)

13 J. M. Rehg © 2002

Color RetinexColor Retinex

Images courtesy John McCann

14 J. M. Rehg © 2002

Finding SpecularitiesFinding Specularities Dielectric materialsDielectric materials

Specularly reflected light has the color of the sourceSpecularly reflected light has the color of the source Reflected light has two components, we see their sumReflected light has two components, we see their sum

Diffuse (body reflection)Diffuse (body reflection) Specular (highlight)Specular (highlight)

Specularities produce a “Skewed-T” in the color Specularities produce a “Skewed-T” in the color histogram of the object.histogram of the object.

15 J. M. Rehg © 2002

R

G

B

Illuminant color

Diffuse component

T

S

Skewed-T in HistogramSkewed-T in Histogram

A Physical Approach to Color Image UnderstandingA Physical Approach to Color Image Understanding – Klinker, – Klinker, Shafer, and Kanade. IJCV 1990Shafer, and Kanade. IJCV 1990

16 J. M. Rehg © 2002R

G

B

R

G

B

Diffuseregion

Boundary ofspecularity

Skewed-T in HistogramSkewed-T in Histogram

17 J. M. Rehg © 2002

Recent Application to StereoRecent Application to Stereo

Motion of camera causes highlight location to change. Thiscue can be combined with histogram analysis.

Synthetic scene:

18 J. M. Rehg © 2002

Recent Application to StereoRecent Application to Stereo“Real” scene:

19 J. M. Rehg © 2002

Finite Dimensional Linear ModelsFinite Dimensional Linear Models

E ii i1

m

rj j j1

n

pk k ii i1

m

rj j

j1

n

d

irj k i j di1, j1

m,n

irjgijki1, j1

m,n

20 J. M. Rehg © 2002

Obtaining the illuminant from Obtaining the illuminant from specularitiesspecularities

Assume that a specularity Assume that a specularity has been identified, and has been identified, and material is dielectric.material is dielectric.

Then in the specularity, we Then in the specularity, we havehave

Assuming Assuming we know the sensitivities we know the sensitivities

and the illuminant basis and the illuminant basis functionsfunctions

there are no more there are no more illuminant basis functions illuminant basis functions than receptorsthan receptors

This linear system yields This linear system yields the illuminant coefficients.the illuminant coefficients.

pk k E d

i k i di1

m

21 J. M. Rehg © 2002

Obtaining the illuminant from Obtaining the illuminant from average color assumptionsaverage color assumptions

Assume the spatial Assume the spatial average reflectance is average reflectance is knownknown

We can measure the We can measure the spatial average of the spatial average of the receptor response to getreceptor response to get

AssumingAssuming g_ijk are knowng_ijk are known average reflectance is average reflectance is

knownknown there are not more there are not more

receptor types than receptor types than illuminant basis functionsilluminant basis functions

We can recover the We can recover the illuminant coefficients from illuminant coefficients from this linear systemthis linear system

r j j j1

n

pk i r jgijki1, j1

m,n

22 J. M. Rehg © 2002

Normalizing the GamutNormalizing the Gamut

The gamut (collection of all pixel values in image) The gamut (collection of all pixel values in image) contains information about the light sourcecontains information about the light source

It is usually impossible to obtain extreme color It is usually impossible to obtain extreme color readings (255,0,0) under white lightreadings (255,0,0) under white light

The convex hull of the gamut constrains illuminantThe convex hull of the gamut constrains illuminant Gamut mapping algorithm (Forsyth ’90)Gamut mapping algorithm (Forsyth ’90)

Obtain convex hull W of pixels under white lightObtain convex hull W of pixels under white light Obtain convex hull G of input imageObtain convex hull G of input image The mapping M(G) must have property The mapping M(G) must have property WGM )(