CCD focal plane array analog image processor SPIE 977.pdf

8/14/2019 CCD focal plane array analog image processor SPIE 977.pdf

1/9

SPIE Real-Time Signal Processing XI Conf.San Diego, CA Aug. 1988(Proc. SPIE 977, paper 32)

CCD focal plane array analog image processorE-S. Eid and E.R. Fossum

Columbia Univer s i ty , Department of Elec t r i ca l EngineeringNew York, New York 10027

ABSTRACTA focal plane array designed for rea l - t ime , genera l -purpose , imagepreprocess ing i s descr ibed . The analog charge-coupled dev ice-basedar ray opera tes in the charge domain and has sens ing , s tor ing , andcomputing capab i l i t i e s . t captures the image data and performs loca lneighborhood opera t ions . The a r ray i s d i g i t a l l y programmable andvar ious image preprocess ing t asks can be implemented. I t uses as ing le ins t ruc t ion , mul t ip le da ta p a ra l l e l a rch i t ec t u re with one

process ing element serv ing four p ixe l s . t can be programmed toperform ID convers ion pr i o r to output . The ul t ra -compac t imageprocessor i s cur ren t ly being fab r i ca ted with a 3-um, double-poly,double-metal process . The 48 X 48 p ixe l a r ray i s projec ted to achievean i n t e rna l throughput as high as 576 Mops with a 54 dB dynamic range9-b i t equivalent accuracy) and 180 um de te c to r p i t c h . The t o t a lpower a i s s i pa t i on i s es t imated to be 12 m o r l e s s . The t o t a l s ize of2the 59-pad chip i s 9.4 X 9.4 mm .1. INTRODUCTION

Real- t ime v i s ion in machines requ i res the synthesis of imaginghardware, computing hardware, and image process ing sof tware . Formobile robots , the system must be por tab le . These mobile robots maybe used in the fu ture for underwater i n spec t ion , a gr i c u l tu r e , spaceexplora t ion , and t r a ns por t a t i on , i n add i t ion to obvious defenseapp l i ca t ions . Thus, the v i s ion system must be l igh tweigh t andlow-power as wel l as having the high throughput necessary forachieving rea l - t ime opera t ion .

Image preprocess ing tasks can be performed using loca l neighborhoodoperat ions on image a r ray pic ture elements pixe l s ) . General ly, thesepreprocess ing funct ions consume the grea t e s t por t ion of t ime r equ i redby the v i s ion process . In the course of e f fec t i ng the preprocess ingt asks , each pixe l may undergo as many as 100 to 500 simple a r i thmet i coperat ions per frame. The frame r a t e may range from 1 Hz in low speedsystems, to 100 Hz in systems opera t ing on a par with human vis ion , to1000 to 10,000 Hz in high performance systems. The number ofoperat ions required in a 100 Hz frame r a t e system with a modest a r rays ize of 100 X 100 pixe ls eas i l y could exceed hundreds of Mops mil l ionoperat ions per sec) . Advances in very h igh speed i n t eg ra ted c i r c u i t s(VHSIC) may al low s e r i a l computing systems to achieve such highthroughput , but a pa ra l l e l computing approach can be used to a l l ev ia t ethe performance requ i rements . Unfor tunate ly , massively pa ra l l e lcomputing systems are presen t ly incompat ib le with the por t ab i l i t y


2/9

needs of a mobile robot system. Furthermore, the advantages ofmassively pa ra l l e l systems of ten are of f se t by the problem of loadingand unloading the pa ra l l e l da ta from a s e r i a l da ta stream a tsu f f i c i en t l y high da ta r a t e s .I t has been proposed to perform the image ~ r e p r o e s s i n g funct ions

in ~ r l l e l on the image plane i t s e l f Fossum and Joseph e ta l . ) . Since the image data a r r ives in a p a ra l l e l manner and i st ransduced to an e l e c t r i c a l form in p a ra l l e l , t seems na tura l toperform spa t i a l ly -pa ra l l e l image preprocess ing on the image plane aswell . Charge-coupled device CCD) s t ruc t u res are wel l - su i ted for sucha system. The analog na ture of the image data can be compactlyrepresented in the charge domain, requ i r ing a s ing le e lec t rode fors to rage . The image data are re f reshed a t the frame r a t e , so thedynamic nature of CCD s igna l rep resen ta t ion i s genera l ly not aconcern. Charge- t ransfer devices a l ready have es tab l i shed themselvesas the technology of choice for image data readout . The d i f f i c u l t yl i e s in the des ign of the charge-domain c i r c u i t s .In t h i s paper , the design of a CCD focal plane array analog imageprocessor i s descr ibed . The IRET ( in rea l - l ime) chip i s designed forrea l - t ime , genera l -purpose , image preprocess ing . IRET i s cur ren t lybeing fab r i ca ted andes t imated performanc arrae of ngements for t e s t ing t areIRET i s descr ibed as wel l . being made. The

2. RR Y ARCHITECTUREIRET was designedde tec to r a r ray of 48process ing elements

emplX 48PEs)oying a s p a t i a l l y p a r a l l e l a rc h i t e c tu r e .p ixe l s and a processor a r ray of 24 X 24are in tegra ted monol i th i ca l ly on IRET.

ATheun i t ce l l of the i n t eg ra ted ar ray c ons i s t s of one PE and a subar ray of2 X 2 pixe ls . The PE i s a se t of var ious computing and communicatingelementE and each PE supports 4 p ixe l s . A pa r t i a l chip l ayouti l l u s t r a t i n g the array of the p-n junct ion photodiode de tec to rs i sshown in Fig. 1. The above choices of the s i z e s of the ar rays are ofexperimental nature and could be genera l i zed to N X N p ixe l s and M X MPEs. The s ize of the subarray of p ixe l s i n the un i t c e l l of thei n t eg ra ted a r ray wil l be n X n, where n = N/M.

This a rchi tec ture has two main fea tures . The f i r s t one i s thein tegra t ion of the sensing and computing elements in the same ce l l .This makes the otherwise d i f f i cu l t t a sk of communicating betweend i f fe ren t elements an easy one. The second main fea ture i s t ha t thear ray i s desiged with a s ing le i n s t ruc t i on , mult ip le data SIMD)a rchi tec ture . In t h i s approach, each PE in the a r ray ca r r i e s out thesame ins t ruc t ion , but on a d i f fe ren t piece of data depending on thepos i t ion of the ce l l in the array . Each PE does not have to beu l t r a - f a s t to achieve rea l - t ime process ing as in the case of s e r i a lapproaches. A modest clock f requency of 25 MHz and a modest a r rays ize of 24 X 24 PEs achieve a t o t a l throughput as high as 576 Mops.

The s i ze of the de tec to r subar ray suppor ted by each PE, n , i s verys i gn i f i can t . Increas ing n enhances the f i l l - f a c t o r , the f r a c t i on ofrea l e s t a t e u t i l i z e d for photodetect ion , and improves the spa t i a l


3/9

.. .. .. .. .. II II II ..... . ..... . ' .. ... . . 0

. .. .. . ..0-

... . .. . .... . - 'Uo - . ~ . . : . - j : _ ~ : + = - - = + : 4 : - 4 : - f - - . j . . : . - - = + - + : : + = - = + : ~ + - - = + = - - = + - - + - - I : : t - - - + - - + - - + - - - t - - f - . . . : . . . . . ~

-.. . kJ... ....... .

-Fig. 1. Chip l ayout (wi th un i t c e l l c i r c u i t r y removed) i l l u s t r a t i n gthe de tec to r a r ray


4/9


5/9

Each un i t c e l l has a cen t ra l bus. As shown in Fig . 2 a l l c i r c u i t s inthe un i t c e l l are connected to the bus through switches .Communication between d i f fe ren t elements in the same un i t c e l l i sachieved through the bus by applying a sequence of c lock s igna l s tothe swi tches . Communication between d i f fe ren t un i t c e l l s i s achievedthrough a t ransce ive r . The t ransce ive r i s a CCD c i r c u i t t ha t can beprogrammed to t ransce ive s igna l s to or from a hor i z on ta l v e r t i c a l ordiagonal neares t neighbor uni t ce l l . A s e r i a l -p a r a l l e l CCD s h i f tr eg i s t e r i s used to s t ack s igna l s coming from the de tec to r subarray .This b id i r e c t i ona l s t ack can be used for s to r ing purposes .

The un i t c e l l has four CCD computing c i r c u i t s . Two of them ared i f fe rencers s imi la r to t ha t repor ted by Fossum and BarkerS. One ofthem i s gated by a magnitude comparator s imi la r to t ha t repor ted byColbeth e t a l . 6 The four th computing c i r c u i t in the un i t c e l l i s as p l i t t e r s imi la r to t ha t repor ted by Bencuya and Steck1 7 Figure 3shows the l ayout of the un i t c e l l and Fig. 4 shows the un i t c e l lin te rconnec t a rchi tec ture .4. CAPABILITIES

The bas ic funct ions of IRET are : capture of the image da taperforming l oca l neighborhood opera t ions and output of a s e r i a l datast ream tha t represents the processed image. t performs the loca lneighborhood operat ions using basic a r i thmet i c funct ions such asaddi t ion sub t rac t ion s p l i t t i n g and magnitude comparison. t a l souses condi t ional addi t ion and sub t rac t ion t ha t i s addi t ion andsub t rac t ion condi t ioned on magnitude comparison.IRET i s a genera l -purpose image processor . t can be programmed toimplement var ious image preprocess ing t a sks each i s a convolut ion of

the image da ta a r ray with a kerne l . The kernel has a c e n t r a l elementand some surrounding ones. The shape and s i z e of the kernel mayd i f f e r depending on the nature of the t ask .Level sh i f t i ng and gain adjus tments are examples of s imple t a sks .In the f i r s t t a sk the cen t ra l pixe l of the kernel i s s h i f t e dpos i t i ve ly or nega t ive ly by a f ixed l e ve l while in the second one iti s scaled by a f ixed f r ac t i ona l fac tor . Other examples inc ludethreshold ing or b i - l eve l coding smoothing or weighted averaging andsharpening or emphasizing the d i f fe rence between the c e n t r a l p ixe l andthe surrounding ones. Edge de tec t ion i s an example of a moresoph i s t i ca ted t a sk t ha t can be implemented.Another family of image preprocess ing t a sks i s the f rame-to-framese t of t a sks . Frame-to-frame operat ions can be performed in a s imi la rmanner. n example i s a motion detec t ion a lgor i thm which can beimplemented using f rame-to-frame di fference opera t ion .IRET can be programmed to perform ID convers ion which i s thegenera l i za t ion of b i - l e v e l coding. ID convers ion can be performedpr io r to output . In t h i s case the output of IRET wi l l be d ig i t a l .


6/9

.

OElECTOft SUB RR Y

Fig 2 Block diagram of the un i t c e l l

Fig 3 Layout of the un i t c e l l


7/9

Fig 4 Uni t c e l l in te rconnect a rch i t e c tu r e


8/9

5. PERFORMANCEIRET i s cur ren t ly being fab r i ca ted with a 3-um, double-polys i l icon ,double-metal process by a commercial CCD foundry se rv ice . Thepred ic t ion of the performance of IRET i s based on numerical analys isand experimental r e su l t s of t e s t i ng a proto type charge-coupled

computer 3 The assumption t ha t the CCDs wi l l opera te with 40 nseccha rac t e r i s t i c clock widths i s a conservat ive one. At most, 25 clockcycles are needed to execute any of the basic opera t ions . Thus, thethroughput of each PE i s es t imated to be 1 Mops and the t o t a lthroughput of the 24 X 24 PE a r ray i s es t imated to be 576 Mops. Onthe average, 250 operat ions per p ixe l per frame are needed toimplement any of the image preprocess ing t asks descr ibed above. Sinceeach PE supports 4 pixe l s , 1000 operat ions per PE per frame areneeded. The 1 Mops per PE throughput yie lds a frame r a t e of 1000frames per sec. This frame ra te i s high and covers a wide range ofapp l i ca t ions . However, IRET wi l l not operate a t t h i s high frame r a t ebecause of the r e l a t i ve l y l imi ted handl ing capab i l i t y of the outputs h i f t r eg i s t e r and ampl i f i e r . t wi l l opera te a t a nominal frame r a t eof 50 frames per sec, so the nominal c lock f requency wi l l be 1.25MHz The t o t a l throughput a t the nominal poin t of opera t ion wil l be28.8 Mops.The de tec to rs are expected to have a 60 dB dynamic range l a -b i tequivalent accuracy) . At 250 operat ions per p ixe l per frame, thet o t a l noise assoc ia ted with computation i s of the same order ofmagnitude as the noise associa ted with sens ing the image data shotno i se ) . Thus, the overa l l dynamic range i s es t imated to be 54 dB9-b i t equivalent accuracy) .CCDs are designed to have a maximum s igna l o f one mil l ion e l ec t rons

per pixe l . On the average, they are es t imated to consume 0.8 pJ pert r ans fe r a t 10 V clock vol tage swing. At the nominal operat ing poin t ,the clock frequency i s 1.25 MHz and the 24 X 24 PEs are expected tod i s s ipa t e 0.6 mW. At fu l l operat ion , the c lock f requency i s 25 MHzand t o t a l power d i s s ipa t ion i s es t imated to be 12 mW.6. CONCLUSIONS

The design of a CCD focal plane ar ray analog image processor chipIRET) has been descr ibed . The capab i l i t i e s of IRET to implementvar ious rea l - t ime image preprocess ing t asks have been t heo re t i ca l l ydemonstra ted and i t s performance has been pred ic t ed . I t fea tu res highthroughput , programmabil i ty , compactness, and low power d i s s ipa t ion .7. ACKNOWLEDGMENTS

The authors wish to gra t e fu l l y acknowledge discuss ions with R.E.Colbeth, S.E. Kemeny, and J - I . Song of Columbia Univer s i ty and theass i s t ance of Dr. R. Bredthauer of Ford Aerospace, Newport Beach,Cal i fo rn ia , i n the fabr ica t ion of IRET. This work was supported by anNSF Pres ident ia l Young Inves t iga tor Award and the NSF Center forTelecommunicat ions Research a t Columbia Univers i ty .


9/9

8. REFERENCES1 . E.R. Fossum, Charge-Coupled Analog Computing Elements and TheirAppl ica t ion to Smart Image Sensors , Ph.D. t he s i s , Yale Univer s i ty(1984).2. J .D. Joseph, P.C.T. Rober t s , J .A. Hoschet te , B.R. Hanzal , and

J .C. Schwanebeck, "A CCD-based p a r a l l e l analog processo r , inSt a t e -o f - t h e -A r t Imaging Arrays and Thei r Appl ica t ions , K.N.Pret ty johns , ed . , Proc. SPIE 501, 238-241 (1984).3. E.R. Fossum, Charge-coupled computing fo r foca l plane imagepreprocess ing , Opt. Eng. 26(9) , 916-922 (1987).4. E.R. Fossum, Charge-domain analog s igna l process ing forde tec to r a r rays , to be publ i shed in Nuclear Ins t ruments andMethods A.5. E.R. Fossum and R.C. Barker , "A l i ne a r and compactcharge-coupled charge packet di f fe rence r r e p l i c a t o r , IEEE Trans.Elect ron Devices ED-31(12) , 1784-1789 (1984).6. R.E. Colbeth, N.A. Doudoumopoulos, E-S. Eid, S .E. Kemeny, A.Montalvo, and E.R. Fossum, High performance charge packet comparisonfor analog D appl ica t ions , Columbia Univer s i ty , TR Tech. Rep.(1987).7 . S.s . Bencuya and A.J . Steck l , Charge packe t s p l i t t i n g in chargedomain dev ices , IEEE Trans. Elec t ron Devices ED-31(10), 1494-1501(1984).

CCD focal plane array analog image processor SPIE 977.pdf

Documents