First Electro Optical System of the 21st Century

8/9/2019 First Electro Optical System of the 21st Century

1/8

PROCESSING REQUIREMENTS FOR THE FIRST ELECTRO OPTIC

SYSTEM OF THE TWENTY FIRST CENTURY

William F. O'Neil

Northrop Grumman Corporation

Electronic Sensors and Systems Division

0NEIL.W.FBPOSTAL.ESSD.NORTHGRUM.COM

ABSTRACT

The Distributed Aperture Infrared System (DAIRS)

defines a new direction in the design of electro-optic

systems. A multiplicity of identical sensors provides

the data for a central processing system. A variety

of signal processing algorithms are used to obtain

the functionality that was previously obtained using

specialized sensors for each function. This permits

a single system to be a Missile Warning set, an

Infrared Search and Track set, the Pilotage sensor,

and a Situational Awareness system as well as

performing various tasks associated with the

warfighting mission of the platform. This paper

describes the DAlRS system being evolved for the

Joint Strike Fighter (JSF) which will be the first

US

aircraft developed in the 21 century. The system is

uniquely processor driven, representing a transition

in electro-optics that parallels the transition in radar

systems that occurred

20-30

years ago. The

algorithms used to implement the functions are

briefly described. The paper presents processor

loading,

I/O

and memory requirements peculiar to

the JSF. Application of the DAlRS technology to

other platforms and applications is also discussed.

A short speculative section on possible future trends

concludes the paper.

BACKGROUND

The development of electro-optic sensors for both

the visible and infrared IR) spectra has been a

major thrust of military systems for the last thirty

years. The third generation of

IR

sensors is now

becoming available. These are two-dimensional

arrays of detectors providing as many as one million

detectors in a single sensor. These sensors alter

the economics of IR sensing in the same way that

the Silicon CCD changed the visible sensor market.

A basic IR sensor can now consist of optics, the

detector array, and some simple electronics. For

the highest performance systems, a cryogenic,

closed cycle cooler

is

still required. The emergence

of room temperature IR detectors will further simplify

the sensors and reduce the cost. These new

sensors provide an opportunity to rethink the design

of E-0 systems. At Northrop Grumman, we have

been pursuing the development of new system

concepts [1,2] since the first staring sensor concepts

were being investigated. DAlRS is a program

funded jointly by Northrop Grumman and the US

Navy Naval Air Warfare Center to bring a distributed

aperture system to flight test status.

DAIRS CONCEPT

ThreaWisslle

Wsrni

Figure 1 DAlRS Provides

4n

Steradian Sensor

Coverage and Multiple Functions

DAlRS (Figure 1) uses an array of sensors,

strategically located around the aircraft to provide

47c

sensing for missile threat warning, IR aircraft search

and track, situational awareness, pilotage, battle

damage assessment, and weapon delivery support.

These sensors are fixed to the aircraft which avoids

the high cost of a mechanism for pointing and

stabilization. DAlRS provides the functions

previously assigned to separate sensor systems with

a single system. A typical sensor design (Figure 2)

is less than five percent of the weight, volume and

power

of

current airborne

IR

sensors. A suite of six

sensors is the minimum configuration for full

coverage due to limitations in the design of optics

with a field of view of more than about 90x900.

0-7803-4150-3/97 10.00 997

l

5.1 15
http://postal.essd.northgrum.com/http://postal.essd.northgrum.com/


2/8

Figure

2

Projected DAldS Production

Configuration fits in

150

in3 volume

The functions that DAlRS provides are enhanced if

the resolution is as fine as the optics will permit.

Current wide field of view optic designs can support

detector arrays with as many as 4Kx4K detectors

which is well beyond the near term detector state of

the art. For the Joint Strike Fighter

(JSF)

which is

the next opportunity for an advanced

E-0

system,

the largest arrays that will be available for the E&MD

phase will have l K x l K detectors which have begun

the development cycle. When JSF enters

production in 2006, these arrays will be considered

standard COTS items.

To meet the pedormance goals of the

JSF,

DAlRS

includes a substantial signal processing component

that will increase the effective resolution of the

sensors by at least

2X.

If sufficient processing

power is available, a 4X resolution increase is

possible within the physical limits of optics and

detectors. Figure

3

[3]

illustrates the resolution

enhancements that have been demonstrated using a

technique called microscanning.

b

4:l microscan

Figure

3

M ~ c r ~ ~ ~ ~ n n ~ n gncreases Effective

Resolution to Optical Limit

The results shown in Figure 3were obtained using a

Maximum Likelihood procedure that combined the

samples from sixteen images, then estimated

underlying sample values on a sampling grid w

was four times denser in each direction than

original sampling grid. The estimation process

iterated about fifty times

to

obtain converg

The procedure required hundreds

of

operation

output sample, and there are 16 output sample

input sample.

A

technique has been develop

Northrop Grumman that requires less

than

operations per output sample. Determinati

line of sight motion from a moving platfor

necessary and requires about 80 operations

input sample.

In addition

to

resolution enhancement, DAlR

addressing the non-uniformity correction (NUC

the detector outputs. Non-uniformity from det

to detector

is

a function of operating cond

including cryogenic temperature, scene tempera

atmospheric conditions as well as inh

differences between detectors including lea

currents, readout linearity and unit cell sele

switches. NUC

is

central to achieving max

range performance for the Missile Warning and

RST

functions. NUC, when combined with tem

frame integration can provide as much as a

improvement in sensitivity (Figure 4)[4]. The N

Equivalent Temperature Difference is plotted ag

background temperature under laboratory cond

to illustrate one aspect

of

the NUC problem.

constant offset term is coherently integrated w

temporal integration is used. For the two-

calibration scheme used in this COTS cam

temporal integration is only effective at

calibration temperatures, with offset e

dominating the noise results elsewhere.

1

1 I

I

I

E I I I I

I I I

I

0.1

0.01

O . O O I

15

20 25

30

35

ackground Temperature- O C

Figure 4- NUC Enables Temporal Integratio

Improve Sensitivity

NUC is included in the microscan operation by u

the microscan

to

determine the spatial s

derivative which is then integrated to recove

5.1 16


3/8

scene. This operation suppresses the offset error

which is the important limiter of sensitivity. Gain

errors are both more stable and also less critical in

limiting the detection of low signal to noise targets.

Gain corrections are performed using factory

calibrations and are assumed constant over sensor

life. NUC computation cost is operations per input

sample. Temporal integration requires

4

operations

per output sample.

Providing effective temporal integration on a dynamic

platform requires that successive frames be

precisely registered. Misregistration causes

smearing of the image with a loss of high spatial

frequency detail. The combination of resolution

enhancement, NUC and temporal integration for

sensitivity enhancement requires that the image

motion must be determined to a small fraction of the

smallest scene details to be recovered. The motion

resolution required is in the range of 0.01 to

0.05

detector instantaneous field of view or about 10-50

PPM of the sensor field of view for a sensor with a

1Kx lK focal plane. The problem of registering

successive images has been extensively studied

[e.g. 5, 6, 71.

DAlRS Applications

DAlRS applications can be classed broadly as

imaging applications where the objective is to

maximize the detail, and detection of unresolved

targets such as missiles or aircraft at long ranges,

where sensitivity and suppression of interfering

clutter are the primary objectives. The imaging

applications include pilotagehavigation, situational

awareness, context support for targeting and battle

damage assessment. The unresolved target

applications include missile threat warning and IRST.

Imaging Applications

For pilotage applications, the interaction of DAlRS

with the display system and pilot determines the

requirements. Resolution at unity magnification is

needed to provide the pilot with an out the window

environment that is sufficient for operation

equivalent

to

VFR conditions. Based on experience

with navigation

IR

sensors currently in use, a

resolution of

0.5-1.0

milliradians appears

to

be the

minimum acceptable range. Achieving human

resolution limit (0.15 milliradians) may not be

necessary due to degradations associated with

aircraft vibration effects

on

pilot vision. A head

mounted display (HMD) is very desirable to

overcome the limits of cockpit display field of regard.

To avoid artifacts when using the HMD, it is

important to have a seamless image with little or no

distortion. A novel mode that will be provided with

DAIRS and an HMD is the ability to see through the

floor when landing a JSF in the VSTOL mode. In

addition, the pilot will be able to view nearby aircraft

for increased situational awareness using an

electronic steerable rear view mirror.

If the HMD is a transparent type that permits seeing

the actual scene with the sensed scene overlaid,

then precision registration is also mandated. The

planned approach for DAlRS is to remove all

distortions by projecting the pixels onto a standard

grid that is aligned with the aircraft IMU

to

a precision

of a fraction of the output sample resolution. This

requires that image warping algorithms operate in

real time (6 operations per output sample). Since

the sensors will be moving relative

to

the IMU due to

aircraft flexing in flight, the alignment properties must

also be calculated in real time. Rate sensors at

each DAIRS sensor are provided for that purpose.

The standard grid also provides registration between

the DAlRS image and the magnified image that is

provided by the targeting sensor. Insetting a portion

of the magnified image into the unity magnification

image provides the context for the magnified image

to aid the pilot in acquisition and identification of

targets.

Battle damage assessment (BDA) is best done with

a very high resolution sensor. However, DAlRS can

provide useful data and compensates for its limited

resolution by providing continuous, all-aspect

coverage. Events where time history is an important

clue include impact explosions, delayed detonation

weapons that produce an ejected plume of hot gas

or debris, and targets that burn after weapon impact.

For these cases, DAlRS is a functional BDA sensor

that can replace high resolution sensors. The key

requirement is sufficient dynamic range to permit

sensing these high intensity but brief events without

the overload and long recovery time of presently

fielded systems. Novel control strategies are

anticipated to optimize the use of DAlRS for BDA.

They will serve to augment the dynamic range of

60-

70

dB available on an intrascene basis

to

achieve an

interscene dynamic range greater than

90

dB.

Unresolved Target Applications

For

unresolved targets the issues are similar for both

missile warning and IRST. The maximum

achievable range at which a targethhreat can be

detected is determined by sensitivity. This is known

5.1 17


4/8

as the clear sky range and is inversely proportional

to detector instantaneous field of view (IFOV).

Temporal integration increases this range as the

fourth root of the number of frames of integration

used. Since the target has not yet been detected,

and will usually be moving with respect

to

the inertial

background, direct target registration is not feasible.

Instead, an array velocity filter is used where

integration is performed for a range of target line of

sight (LOS) rates. For missile targets, the threats

are limited to objects that have a near zero LOS rate.

For aircraft targets, long range detection also limits

the LOS rate while advantage can be taken of the

low vertical range of angles and rates of climb

possible. Thus, while velocity filtering can be

computationally ntensive, the problem can be limited

by the physicsof each engagement.

The clear sky range is frequently of academic

interest because the available range is limited by

false alarms due to clutter. While

a

signal to noise

ratio of

3-5

might suffice for an acceptable false

alarm rate

for

a clear sky target, anecdotal reports

indicate that a typical aircraft against an urban clutter

background may require that the threshold for signal

to noise ratio be increased

to

50-100 to achieve an

acceptable false alarm rate. The resulting drop in

range

3-6x)

is dramatic, and nullifies the benefits of

a practical sensor design. Similar effects have been

noted for the missile threat case. Clutter

suppression is the key issue in the detection of

unresolved targets. The available strategies for

clutter suppression include spatial filters, track file

filters, velocity filters, and multi-spectral filters.

The characteristics of ground clutter are quite

variable. Measurement reported in [8] how a

general tendency to reduced clutter amplitude with

increased spatial frequency. This translates to

improved clutter suppression with increased sensor

resolution (e.g. Figure 5which uses the measured

trends from

[a]

for urban clutter). Spatial filtering is

used to exploit this characteristic as a first step in

most clutter suppression algorithms.

1

P

a

0

0.1

E

a

5

.01

.-

m

K

.

c

0 00

100 1000 10

Array Size Elements

Figure 5 Clutter Amplitude is Decreased by

Reducing Sensor IFOV (90

x

90 FOV)

Track file filters are the simplest clu

discriminators. The successive locations of

potential threat object are associated, and

resulting track is subjected to a variety

reasonableness tests. The processing loads

modest because the track file system is a p

detection system. To limit the number of candid

objects, the threshold is normally adjusted usin

CFAR method. The most difficult problem for tra

file filters is the association problem of link

observations from consecutive frames. Associa

is normally performed using a temporal projec

that defines a window in the next frame that sho

contain the track file object. The accuracy of t

projection depends directly on the sensor resolu

which favors a higher resolution sensor.

Velocity filters operate on the entire scene since th

are pre-detection filters. The ability to discrimin

velocity is the key

to

success in this method. Sinc

is relatively easy to determine the

LOS

rate for

inertial background (terrain), velocity filtering is m

effective in separating threats from terrain. T

limiting feature is the velocity resolution (Figure 6)

0 2 4 6 8 10 12 14 16 18

Range Km

Figure

6

Increased Resolution Increases the

Discrimination Range of Velocity Filtering

5.1 18


5/8


6/8

pixel, the baseline six sensor DAlRS system

operating at 30 Hz frame rate requires about 29-42

billion arithmetic operations per second. Operation at

60

Hz, will double the required operations.

Resolution

Sensitivity

Defensive IRST

The memory requirements are dominated by the

RE

algorithm. The input image is expanded

to

the high

resolution format using two gradient images and

other intermediate results. For 2:1 resolution

enhancement, the total memory storage is 17 pixels

per input pixel. Thus each 1024x1024 sensor

requires 17 megawords 34 megabytes) of storage.

If we can provide processing to achieve the 4:l

resolution enhancement that the optics can support,

the required storage grows

to

65 pixels per input

pixel

or

130 megabytes of storage per sensor.

7.5 BOPS

0.6 BOPS

1.25-15 GFLOPS 1-2 MByte

The IRST algorithms have a range of requirements

between 29 (optimistic) and 2,000 (very pessimistic)

operations per pixel A reasonable estimate is 50-100

operations per pixel. However, there is a very real

probability that the required frame rate can be

reduced

to

5-10 Hz by skipping frames without

significant impact on Pd or Pfa. Also, it is almost

certain that use of off-board data can also reduce

this number.

The resulting load will range from 1.5

to 18 GFLOPS for the IRST algorithms. These are

the pixel level algorithms and do not include the track

file, IMU interface and sensor management

algorithms. While those algorithms will represent the

largest effort in software development, they will have

negligible impact on the throughput requirements. A

typical estimate would be 1,000 CFAR candidates

per frame, each requiring 10,000 instructions per

candidate which yields a load

of

0.3 BIPS.

Missile threat warning, requires fewer than 10

operations per pixel. Additional operations may

prove beneficial in exploiting the high resolution of

the DAIRS, but this has not yet been established.

30

operations per pixel is the current upper bound

estimate, even with full clutter suppression

processing. Assuming the use of 2:l resolution

enhanced data, the required throughput is 7.5-22.5

BOPS at

30 Hz.

Considering the threat regions that

are not approachable by an attacking missile can

reduce

this

by about one half.

For the sky regions,

there is no benefit to using the resolution enhanced

data. This reduces the rate in that region by

4:l.

This more complex sensor management strategy

reduces the required throughput to 3-9 BOPS.

The resulting throughput and memory requirements

are summarized in the Table

I.

No margins are

applied. All entries are additive.

I DAIRS Processor Requirem

Throughput I Memo

Enhancements I 12 BOPS 170 Mbyte

NUC I 1.2-1.5BOPS

I

Totals 29-42 BOPS

I

175 MByte

Table

I

Summary Processing and Memory

Requirements for DAlRS

Alternate Applications of

DAlRS

In addition to the JSF role, Dairs has also be

studied for use on other platforms (Figure8).

Figure 8

-

DAlRS Candidate Platforms Span th

Range of Military Applications

Existing fixed and rotary wing aircraft can bene

from the multi-function capabilities

of

distribut

aperture systems as a path

to

increased survivabil

Rotary wing aircraft will benefit from high frame rat

due

to

the high scene motion rates for nap-of-th

earth operation.

For armored vehicles, the threat

top down attack requires that an early warni

capability be added to existing vehicles, together w

countermeasures that can prevent weapons fro

reaching the vulnerable top surfaces. In addition, t

ability

to

provide functions equivalent to the existi

commanders viewer, gunners weapon sight a

5.1

20


7/8

driver's viewer while reducing signature and

eliminating gimbaled sensing are all attractive

features. Shipboard defense against anti-ship

missiles and threat aircraft has included vigorous

pursuit of passive E-0 sensors due to their high

spatial resolution compared to

RF

sensors. The

scanned sensor technology previously applied

proved to be heavy and expensive, and placed

claims on mast space that could serve other

functions more efficiently. DAIRS, with its small size

and absence of gimbal mechanisms is projected to

save 60-70percent of the weight and cost of a

scanned system while coexisting with other mast

mounted systems.

Future Trends in

E 0

System Design

Two features will dominate the design of

E-0

systems for the near future:

Increased unctionality, and

Large processing growth.

Application of the resolution enhancements being

developed for DAlRS will permit the design of

gimbaled sensors with simplified optics. The

multiple fields of view needed to optimize search,

detection, recognition and weapon delivery will be

obtained from a single large optical field of view

using processing to achieve the higher resolutions

needed.

Gimbal elimination will remain an elusive goal, but

may be achieved using large aperture DAlRS

designs. This is still an attractive solution because

the gimbal mechanism is the most expensive part of

current

E - 0

sensors. The gimbal is currently

justified by the need for a laser illuminator for laser

guided weapons. If these are replaced by more

covert and self contained weapons, then the

elimination of the gimbal will be practical with

considerable benefit to airborne platforms.

The capabilities of the DAlRS sensors remains

limited by the size of detector arrays.

The small

market base for ever larger arrays will inhibit their

development, particularly if processing can be shown

to achieve the same result using resolution

enhancement techniques. The fundamental

physical limits of optics resolution will also be

overcome using advanced processing methods and

large stored data bases. Efforts in neural networks

can increase resolution. Extending the optical

resolution by even a factor of two is a substantial

benefit. Neural network resolution enhancement

can be performed in a context free mode, but, in the

near term, providing a context basis using stored

data will be an attractive approach requiring

substantially ess computation.

Sensitivity is also limited by detector physics. The

dynamic range of a photovoltaic detector is limited by

its charge storage capacity. The dynamic range

requirements for IR scenes are determined by the

scene equivalent temperature excursions and the

noise equivalent temperature difference (NETD) of

the sensor system. The environment is usually

taken to span an intrascene dynamic range of

80

100 K which results in occasional saturation in non-

combat conditions, and frequent saturation in

battlefield conditions. Current sensors have

achieved NETD near .01 K and efforts continue to

achieve NETD of .001

K.

The required dynamic

range is in the range of lo (14 bits / benign

environment) to lo6 (20 bits

/

battlefield

environment). Available arrays store 10 electrons

per detector which is less than

12

bits dynamic

range. (Dynamic range for high background IR

sensors is limited to the square root of the electron

storage capacity.) Experimental arrays with 2.5E7

stored electrons per detector have been fabricated.

Achieving a

20

bit dynamic range would require

increasing electron storage by

lo5

which is

problematic.

Raising the frame rate and using post readout

integration is a feasible alternative. The result

would be arrays with millions of detectors being

sampled at hundreds of frames per second. The

increased frame rate is attractive because it allows a

more dynamic environment with greater bandwidth

for platform and scene motions, while providing

adaptive response to scene signatures at the pixel

level. Sensor sensitivity and speed of response

becomes a software function which opens the design

trade space to developments that cannot be

foreseen today. Experimental

51

2x51

2

arrays

operating at 480

Hz

have been built. Multiport

arrays with data rates of

lo9

samples per second

with 16 bit dynamic range can be anticipated from

current developments.

Multi-spectral sensors are another area of active

research that will impact data processing

requirements. Current designs range from two

color sequential sensors to multi-spectral sensors

that provide data at a large number of wavelengths

simultaneously. Using existing staring detector

arrays imposes a spatial, temporal and spectral

trade-off in which the num ber of wavelengths, the

array spatial coverage and the time (number of

frames) needed to satisfy a required spatial

coverage are combined to satisfy a mission

5.1 21


8/8

objective. Two difficulties are the registration of

time displaced spectral observations that is required

when all of the spectra are not available

simultaneously

for

the same spatial locations, and

the problem of radiometric conversion of detector

spectral data in the presence of readout errors. The

detector errors are similar to the NUC errors

discussed above, with the addition

of

detector

spectral gain errors. Difficulties in characterizing all

of the errors in a dynamic environment must be

overcome to enable spectral techniques. Methods

currently being developed to determine detector

nonlinear characteristics may be necessary, as well

as methods to establish temporal characteristics.

This could lead to multi-dimensional NUC tables that

would require significant increases in processing

power.

CONCLUSION

Distributed aperture sensor systems are a new

direction in electro-optics. Large staring focal plane

arrays support multiple functions and eliminate the

costly stabilization and pointing gimbals of current

systems. Initial designs are processing intensive

due to high data rates, increasing functions per data

sample and inherent characteristics of arrays with

one million detectors or more. Entry level

processing capabilities are in the range of one

hundred billion arithmetic operations per second with

memory storage of several hundred megabytes.

This provides missile warning,

IR

search and track,

pilotage, battle damage assessment and weapon

delivery support. Distributed aperture systems are

widely applicable to military systems and are likely to

be evolved for multiple platforms. Growth will occur

in both the size and speed of the detector arrays, but

also in the applications that the data will support.

Throughput increases of one

to

two orders of

magnitude will happen when the processing

becomes economically available. These

developments will use image understanding

technologies that are already in place. Continuing

evolution in that area will result

in

even greater

requirements or increased processing.

REFERENCE

[ l]

Hale,

R.

A. et al United States Patent No.

5,317,394 Distributed Aperture Imaging and

Tracking System issued May 31, 1994

[2] Hale,

R.

A. et al - United States Patent No.

5 4 12,421 Motion Compensated Sensor issued

May 2,1995

[3]

R.

C. Hardie and E. Kaltenbacher

-

High

Resolution Infrared Image Reconstruction using

Multiple, Low Resolution, Aliased Frames

-

to be

published

[4] ONeil

W .F,

Experimental Verification of Dithe

Scan Non-uniformity Correction, 1996 meeting o

the

IRIS

Specialty Group on Passive Sensors, March

13,1996

[5] Schaum, A. and McHugh, M., Analytic Methods

of Image Registration: Displacement Estimation and

Resampling, NRL Report 9298, Feb. 28,1991

[6] Hench, D. and Fried, D., Status o

Pseudoregistration Development BC-276, the

Optical Sciences Company, Placentia CA., Feb

1985

[7] Kuglin, C. and Hines, D., The Phase Correlatio

Image Alignment Method.

IEEE

Proc. 1975 Conf. on

Cybernetics and Society, Sept. 23-25, 1975 pp. 163

165

[8] McGlynn, J. D. and Sofianos, D.

J.,

Parametric

Model-based Characterization of

IR

Clutter, Science

Applications International Corporation Repor

(undated)

[9] J. M. Mooney et al, Responsivity Nonuniformit

Limited Performance of Infrared Staring Cameras

Optical Engineering, Nov. 1989,

Vol

28, NO. 11, p

1151

5.1 22

First Electro Optical System of the 21st Century

Documents