4

1

ComputerVision

Recording and Displaying

of Images

Recording and Displaying

of Images

ComputerVision

Acquisition and display of images

1. displays

A reader’s digest :

2. cameras

3. illumination

2

ComputerVision

displays

ComputerVision Cathode Ray Tubes

3

ComputerVision The video standard

viewing quality ⇒ psychophysics

discrete character, both spatial and temporal,should be invisible

video standard defines :

1. number of lines and columns2. aspect ratio3. scanning frequency

ComputerVision TV : number of lines

Aspect ratio of pixels = 4 /3

Europe : CCIR : 625 lines, 575 visibleUSA : RETMA : 525 lines, 484 visible

(other lines lost during flyback)

4

ComputerVision

TV : sampling frequencyflickerfree images : at least 60 Hzhalved by interlacing : odd and even lines separately :

Europe : repetition of 25Hz (50/2)USA : 30Hz (60/2)

1

3

5...

2

4...

1st field2nd field

INTERLACING

ComputerVision LCD display

Liquid Crystal Display

polarizers

passive pixel

active pixel

No light generation ⇒ use backlighting

5

ComputerVision LCD display

Liquid Crystal Display

Electrical field

polarizers

passive pixel

active pixel

No light generation ⇒ use backlighting

ComputerVision

cameras

6

ComputerVision Optics for image formation

the pinhole model :

mZf

YY

XX

oo

i

o

i −=−

==

(m = linear magnification)

ComputerVision Camera obscura + lens

7

ComputerVision

The thin-lens equation

lens to capture enough light :

fZZ iO

111 =−

assuming spherical lens surfacesincoming light ± parallel to axisthickness << radiisame refractive index on both sides

PO

ComputerVision

The depth-of-field

fbdfZfZZZZZ−+

−=−=Δ −−

/ )(

0

00000

decreases with d, increases with Z0strike a balance between incoming light and large depth range

8

ComputerVision Deviations from the lens model

3 assumptions :

1. all rays from a point are focused onto 1 image point

2. all image points in a single plane

3. magnification is constant

deviations from this ideal are aberrations

ComputerVision Aberrations

chromatic : refractive index function of wavelength

2 types :

1. geometrical

2. chromatic

geometrical : small for paraxial rays

9

ComputerVision Geometrical aberrations

spherical aberration

astigmatism

radial distortion

coma

aberrations are reduced by combining lenses

the most important type

ComputerVision Radial Distortion

magnification different for different angles of inclination

Can be corrected if parameters are known

10

ComputerVision Chromatic aberration

rays of different wavelengths focused in different planes

cannot be removed completely

sometimes achromatization is achieved formore than 2 wavelengths

ComputerVision

Cameras

we consider 2 types :

1. CCD

2. CMOS

11

ComputerVision

Cameras

ComputerVision

Cameras

12

ComputerVision

Cameras

ComputerVision The CCD camera

13

ComputerVision CMOS

Same sensor elements as CCDEach photo sensor has its own amplifier

More noise (reduced by subtracting ‘black’ image)Lower sensitivity (lower fill rate)

Uses standard CMOS technologyAllows to put other components on chip‘Smart’ pixels

Foveon4k x 4k sensor0.18μ process70M transistors

ComputerVision CCD vs. CMOS

• Mature technology• Specific technology• High production cost• High power consumption• Higher fill rate• Blooming• Sequential readout

• Recent technology• Standard IC technology• Cheap• Low power• Less sensitive• Per pixel amplification• Random pixel access• Smart pixels• On chip integration

with other components

2006 was year of sales cross-over

14

ComputerVision Colour cameras

• We consider 3 concepts:

1. Prism (with 3 sensors)2. Filter mosaic3. Filter wheel

ComputerVision Prism colour camera

Separate light in 3 beams using dichroic prismRequires 3 sensors & precise alignmentGood color separation

15

ComputerVision Prism colour camera

ComputerVision Filter mosaic

Coat filter directly on sensor

Demosaicing (obtain full colour & full resolution image)

16

ComputerVision Filter wheel

Rotate multiple filters in front of lensAllows more than 3 colour bands

Only suitable for static scenes

ComputerVision Prism vs. mosaic vs. wheel

Wheel1

GoodAverageLowMotion3 or more

approach# sensorsSeparationCostFramerateArtefactsBands

Prism3

HighHighHighLow3

High-endcameras

Mosaic1

AverageLowHighAliasing3

Low-endcameras

Scientific applications

17

ComputerVision Upcoming X3 technology of Foveon

creates a stack of pixels at one placenew CMOS technology

Exploits the wavelength dependent depth to which a photon penetrates silicon

And splits colors without the use of any filters

ComputerVision Geometric camera model

(Man Drawing a Lute, woodcut, 1525, Albrecht Dürer)

perspective projection

18

ComputerVision Models for camera projection

the pinhole model revisited :

center of the lens = center of projection

notice the virtual image plane

this is called perspective projection

ComputerVision Perspective projection

origin lies at the center of projection the Zc axis coincides with the optical axisXc-axis ⎢⎢ to image rows, Yc-axis ⎢⎢ to columnsZ

Xfu = ZYfv =

Yc

Zc

Xc

v

u

19

ComputerVision Pseudo-orthographic projection

ZXfu =

ZYfv =

If Z is constant ⇒ x= kX and y = kY,where k=f/Z

i.e. orthographic projection + a scaling

Good approximation if ƒ/Z ± constant, i.e. if objects are small compared to their distance from the camera

ComputerVision

Pictoral comparison

Pseudo -orthographic Perspective

20

ComputerVision

Projection matrices

the perspective projection model is incomplete :what if :

1. 3D coordinates are specified in a world coordinate frame

2. Image coordinates are expressed as row and column numbers

We will not consider additional refinements,such as radial distortions,...

ComputerVision

Projection matrices

X

Z

Y

0

v

u

(u,v)

C

(X,Y,Z)

r1r3

r2

)()()()()()()()()()()()(

333232131

323222121

333232131

313212111

CZrCYrCXrCZrCYrCXrfv

CZrCYrCXrCZrCYrCXrfu

−+−+−−+−+−=

−+−+−−+−+−=

C,PrC,Pr

fv

C,PrC,Pr

fu

−−

=

−−

=

3

2

3

1

21

ComputerVision

Projection matricesImage coordinates are to be expressed as pixel coordinates

⎩⎨⎧

+ =+ + =

0

0

yvkyxvsukx

y

x

→ (x0, y0) the pixel coordinates of the principal point → kx the number of pixels per unit length horizontally→ ky the number of pixels per unit length vertically→ s indicates the skew ; typically s = 0

with :

NB1: often only integer pixel coordinates matterNB2 : ky/kx is called the aspect ratioNB3 : kx,ky,s,x0 and y0 are called internal cameraparametersNB4 : when they are known, the camera is internally calibratedNB5 : vector C and matrix R∈ SO (3) are theexternal camera parameters

NB6 : when these are known, the camera isexternally calibratedNB7 : fully calibrated means internally and externally calibrated

x

ym

n

0 1 20123

ComputerVision Projection matrices

Exploiting homogeneous coordinates :

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

−−−

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

3

2

1

333231

232221

131211

1 CZCYCX

rrrrfrfrfrfrfrf

vu

τ

We also have

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

11000

10

0

vu

ykxsk

yx

y

x

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

=0

100

01000000

100

0 00

0

ykfxsfkf

ff

ykxsk

K y

x

y

x

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

−−−

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

3

2

1

333231

232221

131211

0

0

1000000

100

01 CZ

CYCX

rrrrrrrrr

ff

ykxsk

yx

y

x

τ

We define the calibration matrix :

22

ComputerVision

Projection matrices

We define

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

= ⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

1

~,;1

ZYX

PZYX

Pyx

p

yielding

)( CPKRp t −=ρ for some non-zero ρ∈ ℜ

or, PtMp ~)|(=ρ with rank M = 3

or, ( )PCRRKp tt ~| −=ρ

ComputerVision From object radiance to pixel grey levels

After the geometric camera model...… a camera model

2 steps:

1. from object radiance to image irradiance

2. from image irradiance to pixel grey level

23

ComputerVision Image irradiance and object radiance

we look at the irradiance that an object patchwill cause in the image

assumptions :radiance R assumed known andobject at large distance compared to the focal length

Is image irradiance directly related to the radiance of the image patch?

ComputerVision

The viewing conditions

ια cos cos 32

0

ZAAAR

AFI

i

l

i

==

α42 cos

fAR l=

24

ComputerVision The cos4 law cont’d

Especially strong effectsfor wide-angle and

fisheye lenses

ComputerVision From irradiance to gray levels

dIgf += γ

Gain

“gamma”

Dark reference

25

ComputerVision

illumination

ComputerVision

Illumination

Well-designed illumination often is key in visual inspection

26

ComputerVision

Illumination techniques

1. back-lighting2. directional-lighting3. diffuse-lighting4. polarized-lighting5. coloured-lighting6. structured-lighting7. stroboscopic lighting

Simplify the image processing by controlling the environment

An overview of illumination techniques:

ComputerVision Back-lighting

lamps placed behind a transmitting diffuser plate,light source behind the object

generates high-contrast silhouette images,easy to handle with binary vision

often used in inspection

27

ComputerVision Example backlighting

ComputerVision

Directional and diffuse lighting

Directional-lighting

Diffuse-lighting

generate sharp shadowsgeneration of specular reflection (e.g. crack detection)shadows and shading yield information about shape

illuminates uniformly from all directionsprevents sharp shadows and large intensityvariations over glossy surfaces

28

ComputerVision Crack detection

ComputerVision

Example directional lighting

29

ComputerVision

Polarized lighting

1. to improve contrast between Lambertian and specular reflections

2. to improve contrasts between dielectrics and metals

2 uses:

ComputerVision

polarizer/analyzer crossedprevents the large dynamic range caused by glare

Polarized lightingspecular reflection keeps polarisation :diffuse reflection depolarisessuppression of specular reflection :

30

ComputerVision

Example pol. lighting (pol./an.crossed)

ComputerVision

Polarised lightingdistinction between specular reflection from dielectrics and metals;works under the Brewster angle for the dielectricdielectric has no parallel comp. ; metal doessuppression of specular reflection from dielectrics :

polarizer/analyzer aligneddistinguished metals and dielectrics

31

ComputerVision

Example pol. lighting (pol./an. aligned)

ComputerVision Coloured lighting

highlight regions of a similar colour

with band-pass filter: only light from projected pattern (e.g. monochromatic light from a laser)

differentiation between specular and diffuse reflection

comparing colours same spectral composition of sources!

spectral sensitivity function of the sensors!

32

ComputerVision Example coloured lighting

ComputerVision

spatially or temporally modulated light pattern

Structured and stroboscopic lighting

Structured lighting

e.g. : 3D shape : objects distort the projected pattern(more on this later)

Stroboscopic lighting

high intensity light flash

to eliminate motion blur

33

ComputerVision

Stroboscopic lighting