8/11/2019 Smart CCD image sensors for optical metrology and machine vision.pdf
1/4
Smart CCD Image Sensors for Optical Metrology and Machine Vision
T. Spirig, P. Seitz,
0
Vietze
Paul Schrerrer Institute Zurich
Badenerstrasse 569, CH-8048 Zurich
e-Mail: Thomas. Spirig psi.ch; [email protected]; Oliver.Vietze@ psi.ch
Abstract -Tw o types of CCD image sensors are described.
The first sensor is a two-dimensional, synchronous
detector/demodulator (Lmk-In CCD) of spatially
modulated light fields for applications in heterodyne
interferometry
and
time-of-flight range imaging.
Simultaneous measurements of amplitude, phase and
background level are carried out at e ach pixel site. This
is made possible by the principle of synchronized,
periodic multi-tap sampling and photo charge
accumulation. The second sensor, the Convolver CCD
is capable
of
performing image acqu isition and real-time,
parallel convolution with an arbitrary kernel. Tap weight
accuracies of typically 2 of the largest tap values have
been obtained
for
a variety of linear filters that are
commonly
used
in machine vision. The sensors have been
realized by using a comm ercially available multi-project
wafer CMOS/CCD process. For both sensors, the
principle, design, operation and measurement results are
presented and discussed.
I. INTRODUCT ION
Charge-coupled devices (CCDs) are used in most of
todays image sensing applications, although the primary
motivation for their development was charge storage [I].
Soon, it became apparen t that the CC D principle is also well
suited for the processing of analog signals in the charge
domain [2]. Using modem semiconductor technologies, it is
possible to fabricate photosensors whose geometry and
functionality are adapted to specific sensing tasks of various
optical measurement techniques. In the present work,
two
types of such sensors are presented. The first type, called
lock-in CCD, is a
two
dimensional array of pixels, each of
which is a synchronous detectoddemodulator for spatially
modulated light fields. This is made possible by a
synchronous photo-charge detection a nd storage s proposed
in an incomplete form with
a
modified CCD in Ref. [3]. Th e
Lock-In CCD is described in Section
2.
The second type of
sensor, described in Section 3, is an image sensor capable
of
carrying out convolutions with arbitrary kemels during the
exposure. The possibility of realizing such a device
was
initially suggested by Beaudet [4] oth sensors have been
realized using Orbits Foresight 2pm N-well CMOYCCD
process [ 5 ] In the concluding Section 4 the obtained results
are discussed and possible applications are presented.
11. LOCK-IN CCD
In typical image sensing applications a scene is
optically projected onto the image sensor, which has to
detect the local light intensity of an essentially stationary
scene. There are circumstances, however, in which the
sought information is not the local light intensity, but is
encoded in a modulation parameter of an oscillating
waveform. One might be interested in the local phase of the
modulated light, the amplitude and the background
illumination. The well-established measurement technique
by which local phase, amplitude and offset of a sinusoidal
electrical signal are measured, is called lock-in technique.
Conventional synchronous detectors (lock-in detectors),
often based on phase-locked loop (PLL) technology, are
essentially restricted to the analysis of one temporal signal at
a time. If modulated light is de tected with 3 or more samples
per period, its mean brightne ss level B (offset), phase cp and
amplitude
A
can
be
determined unambiguously. Figure
1
illustrates the measurement principle for the
so
called four-
bucket technique. Th e charge is integrated during time
intervals I1 to I4 of equal length At within one modulation
period T. Four charge packets per pixel, o to a3 are then
obtained, which have to
be
stored at four different spatial
locations. The charge signals a0 to a3 are converted into the
t
T
Fig.
1
Measurement principle
of
the Lock-in CCD. During one
modulation period, four charge packets *a3 are created
by
integrating the photo-generated charge. With these
four
packets
the phase
9
he offset B and &he amplitude can be determined.
(0-7803-2943-0)
11
8/11/2019 Smart CCD image sensors for optical metrology and machine vision.pdf
2/4
sought amplitude
A,
phase cp and mean brightness level B
by using the relationships, described in Reference
[6] .
The architecture of one pixel is illustrated
schematically in Figure 2. One pixel consists of a photo-gate
(PG),
a
dump-gate (DG), a dump-diffusion (DD) and
4
transfer gates (TGo-TG3). In our implementation the area of
the photogate
is
10.5p m x 41.5 pm. Two vertical four-phase
CCD lines are located adjacent to each pixel. The four
transfer-gates act as sw itchable connections between PG and
the four-phase CCD. The four-phase CCD is covered by a
metal layer in order to prevent charge smear during read out.
Tests have been carried out using a n image sensor with
10
x
fo
Fig.
2.
Schematic layout of one pixel.
It
consists
of a
photo-gate
(PG), four transfer-gates (TGo-TGs),
a
dump gate (DG) and
a
Dump diffusion (DD). The charge packets are stored under the
gates fp of the two adjacent CCD lines.
15
pixels, with a horizontal pitch of 92.75 pm and a vertical
pitch of
80 pm.
The op eration occurs in
two
stages, signal integration
and storage,
followed by read out. During the signal
integration phase, the modulated light intensity produces a
modulated photo-current, which is integrated on the
MOS
capacitor of the photosite. Synchronously with the master
oscillator this integrated charge is transferred and
accumulated in one of the shielded storage sites by
appropriately clocking the transfer gates and the photogate.
Th e exposure time and the location where the signal charg e
is stored can be chosen by programming an appropriate
clocking sequence. The pixel can be cleared, e.g. for
shuttering operations, by pulsing the dump -gate high and
the photo-gate low. The charge is then transported to the
dump diffusion.
In the read out phase, the electrodes of the line
transfer CCD are clocked sequentially, transporting the four
charge packets per pixel
in
shielded CCD lines to the output
stage. The signal charge is converted to a voltage by use of a
floating diffusion output stage with a single on-chip p-
channel
MOSFET
acting
as
a source follower.
The CCD has been tested using a light source with a
modulated intensity. The center wavelength is a t 630 nm
and
the beam was focused on the CCD
as
described in Reference
[7] in m ore detail. The m odulation frequency was chosen to
be
100.4
kHz.
This is
a
typical frequency for the intended
application of the lock-in CCD in heterodyne interferometry.
The charge was collected over
100
modulation periods.
Figure 3 shows the measured phase vs. the true master
oscillator phase for 12 different phases starting at
(pmaster = 0
and increasing with a step size of 30 . An individual offset
for all outputs per pixel was subtracted and the resulting
signals were normalized. The r.m.s deviation of measured
vs. true master oscillator phase was
3.1
degrees (relative
phase error of
0.009).
The relative deviation from true
amplitude was
0.05.
C 0 81
measured values
l
average
master osc illator phase [deg]
Fig.
3.
Amplitude measurements obtained with the Lock-in pixel
CCD, exposed to a sinusoidally oscillating LED light source. The
modulation frequency
was 100.4
kHz. The relative amplitude
measurement deviation from the true master oscillator amplitude
was 0.05.
LI. CONVOLVER CCD
any signal and image processing functions are
linear, i.e. one-or two dime nsional convo lutions with specific
kemels
[8].
The two-dimensional result
of
the convolution is
determined
s
a linear combination of neighboring pixel
intensities with suitable weights, as described in Reference
An interpretation of the convolution is that the
picture is shifted laterally in both dimensions and for each
position the pixel values are multiplied with a different
weight and accumulated to the sum, resulting in the new
pixel value.
As
the CCD is capable
of
shifting charge
laterally in two dimensions, this interpretation leads to a
naturally parallel implementation of the convolution using a
CCD.
The
weighting can be realized by varying the exposure
times for the different lateral
shifts.
This principle holds
only for kemels with positive weights. The problem with
negative weights can be solved by performing two
convolutions each with only positive weights and performing
the difference either on or off-chip. The convolution CCD
consists
of
bidirectional CCD columns and bi-directional
rows and at the intersections of rows and columns the
~71.
12
8/11/2019 Smart CCD image sensors for optical metrology and machine vision.pdf
3/4
photosites are located. Each pixel has its
own
associated
storage area, realized as an additive CCD column with
separate gates in parallel to the main CCD column. In our
implementation, the area of the photo-gates is
30
x
30
pm2
and the pixel pitch is
63 pm
horizontally and 65 pm
vertica lly leading to a fill factor of
22 .
First tests have been
performed w ith
16
x 16pixels.
The Point spread function (PSF) can
be
meas ured by
illuminating a single pixel with a
20
pm diameter light spot.
Four types of filters have been chosen, which are commonly
used in machine vision. These are: I)nisotropic Gaussian
filters; (11) Canny filters (the first deviation of Gaussian
filters); (111) Lap lacian of Gau ssian filters (L OG -filters) and
(IV) quadrature
pair
filters (Gabor filters).
A
set of
7 x
7
filter coefficients was calculated for each of these four
continuos functions. They were used to set the exposure
times. As the image is a single spot, there is a direct
correspondence between the calculated coefficients and the
measured ou tput voltage. To obtain a fig ure of acc uracy , the
image was offset corrected and normalized to fit the filter
coefficients and an r.m.s. dev iation is obtained. We found
excellent correspondence between measured values and true
filter values. Typical r.m.s. deviations from the ideal filter
characteristics are between 1-2% of the largest kernel tap
value. As an example the tw o-dimensional point spread
function of a Laplacian of Gaussian filter
is
illustrated in
Figure
4
The following three compo nents contribute to the
error:
(1)
The read-out noise of the output amplifier,
(2)
the
dark current noise, a shot noise compo nent that depends on
the exposure time and
3)
the non-ideal charge transfer
efficiency (CTE). In our case, most of the error is attributed
to the imperfections of ou r measuremen t technique, largely
due to the non-linearity of the output stage. In addition
filters without negative elements in their filter mask have
slightly smalle r r.m.s. erro rs since only one convolution was
Fig.
4
Two-dimensional point spread function of
a
Laplacian
of Gaussian filter measured
with
the convolution CCD. The
r.m.s deviation of the obained point spread
function from the
ideal filter was less than 2 .
performed and hence the expo sure time
for
these filters was
smaller.
ILL DISCUSSION
Two image sensors have been presented with on-chip
signal processing capabilities. These sensors have been
fabricated with a commercially available CM OSK CD
process. The Lock-in CCD accurately measures the phase
and the amplitude of light modulated at 100
kHz
This is
ideal for the intended application in heterodyne
interferometry, higher modulation frequencies are possible,
since the charge coupled transfer principle has been
demonstrated at frequencies up to 32 5 MHz in silicon [9].
Interesting applications for the convolver CCD are in
real time object recognition using m atched filter techniques.
By providing two or more storage sites per pix el, it would be
possible to perform two convolutions and hence to detect
keypoints in real-time
[lo]
Either sensor could
also
find
application in motion detection due to its capability of
storing two (convolver CCD)
or
four (Lock-in CCD)
successive pictures before readout. It is noted that spatial
resolution of the CCDs, i.e. the modu lation transfer function
MTF) is degraded due to carrier diffusion in the
semiconductor sensor
Ell],
[12]. Light with a long
wavelength produces a large penetration depth of the
photons in
the
semiconductor, leading to a larger cross-talk
and hence a reduced spatial resolution for the convolver
CCD and a significantly reduced dynamic range for
measurements with the lock-in CCD .
The presented CCD structures offer an interesting
alternative to the active pixel sensors
A P S ) ,
where the
signal processing is performed with a conventional CMOS
circuitry. The CCD approach has the advantage that the
signal processing is carried out in the charge domain,
without any noise contributions from the device other than
dark noise. The novel device structures presented here are
merely an exam ple of possibilities that CCD pro cessing has
to offer in the field of optical metrology and m achine vision.
ACKNOWLEDGEMENT
We gratefully acknowledge the invaluable help from
many people at the Paul S cherrer Institute in Zurich. Special
thanks goes to M.Kuhn, who provided the design and the
realization of the o ptical setup,
ro
P.Metzler, who designed
and fabricated the trapezoidal clock drivers and to
J.M.
Raynor for his invaluable help with analog and digital
electronic problems. We would also acknowledge
stimulating discussions
wt
0.Kubler and F.Heitger at the
Federal Institute of Technology (ETH) Zurich. This work
was supported in part by the Swiss priority programme
Optique, project number
524
13
8/11/2019 Smart CCD image sensors for optical metrology and machine vision.pdf
4/4
REmRENCES
[41
[71
W.S. Boyle and G E Smith, Charge Coupled
Semiconductor Devices, Bell Syst. Tech. Jour. Vol.
C.H. Shuin and M.F. Tompsett, Charge Transfer
Devices,
Academic Press
Inc., New York, 1975.
H.Pove1, H. Aebersold and J.O. Stenflo,
Chargcoupled device image sensor as a demodulator
in a 2-D polarimeter with a piezoelastic modulator,
Applied Optics
Vol. 29, pp. 1186-1190, 1990.
P.R.Beaudet, Time and Space Multiplexing Focal
Plane Convolvers, Proc. SPZE Vol. 107
1
pp. 90-99,
1989.
Orbit Semiconductor Inc., 1230 Bordeaux Drive,
Sunnyvale, California, 94089, USA.
T.Spirig, P.Seitz, 0 Vietze and F.Heitger, The
Lock-In CCD - Two-Dimensional Synchronous
Detection of Light, submitted to
IEEE J Quantum
Electronics in print (1995).
P. Seitz, T. Spirig, 0 Vietze and K. Engelhardt,
Smart sensing using custom photo-ASICs and CCD
technology,
Optical Engineering
to appear in
August 1995.
W.K.Pratt, Digital Image Processing, 2nd Edition,
Wiley Sons, New York, 1991.
A.L. Lattes, S.C. Munro e and M.M. Seave r,
Ultrafast Shallow-Buried-C hannel CCDs with Built-
In Drift Fields,
IEEE Electr. Dev. Lett.
ol. 12, pp.
F. Heitger, L. Rosenthaler, R. von der Heydt, E.
Peterhans and 0 Kubler, Sim ulation of neural
contour mechanisms: From simple to end-stopped
cells,Vision Research
32, pp. 963-981, 1992.
D.H.Seib, Carrier diffusion degradation
of
modulation transfer function in charge coupled
imagers, IEEE Trans. Electron Devices vol ED-21,
E.G. Stevens and J.P. Lavine, An analytical,
aperture, and two-layer carrier diffusion
MTF
and
quantum efficiency model for solid-state image
sensors, IEEE Trans. Electron Devices vol. 41, pp.
49, pp. 587-593, 1970.
104-107, 1991.
pp. 210-217,
Mar.
1974.
1753-1760, Oct. 19 94
14