Recognition System Capacity

Recognition System CapacityJoseph A. O’Sullivan

Samuel C. Sachs ProfessorElectronic Systems and Signals Research Laboratory

Department of Electrical and Systems EngineeringWashington University in St. Louis

(314) 935-4173; http://essrl.wustl.edu/~jao [email protected]

Michael D. DeVore, UVANaveen Singla

Brandon WestoverSupported in part by the Office of Naval Research

Adaptive Sensing MURI Review, 06/27/06

http://essrl.wustl.edu/~jao

mailto:[email protected]

2

O’Sullivan AS-MURI 2006

Recognition System Capacity

• Motivation: – ATR; Network Centric Warfare;

Biometrics; Image Understanding• Active Computations• Achievable Rate Regions

– Inner and outer bounds– Successive refinement

3


Why Theorems?• ONR Perspective: Want Systems

That Work– Implementable on projected system

architecture– Good performance

• Our Perspective: Theorems Provide– Provable performance: bounds and

guidelines– Validation and critique of existing

system designs– Motivation for recognition system

design: system architectures; database design; optimal compression for recognition; communication for recognition; active computations

• Growing Awareness of Importance of Information Theory

4


Perspective on Image Understanding

• “Finding tanks is so World War II.” Bob Hummel, DARPA program manager, ATR Theory Workshop, Dec. 2004– What make of car?– What year?– Who is driving?– Where has it been?

• Improvised Explosive Devices (IED)• Demand more information from imagery

5


Biometrics Must Be

• Universal• Permanent• Unique• Measurable

Uniqueness How unique? Bits.Measurability How measurable? Bits.

A. K. Jain, et al., “Introduction to Biometrics,” 1999John Daugman, http://www.cl.cam.ac.uk/users/jgd1000/

Encoding

6



• Motivation: – ATR; Network Centric Warfare; Biometrics;

Image Understanding• Active Computations• Achievable Rate Regions


7


Sensor Data

Target Type Estimate

Performance Estimate

Algorithm(ATR)Sensors

Network Resources databases,communications, etc.

System Performance Analysis

ResourceAllocation

Active Computations Concept

Compute a sequence of inferences and performance estimates (probabilities or reliabilities). Monitor available resources (time, processors, bandwidth, database, …). Feed back performance: select next computation; reallocate resources; demand more data.

8


Sensor Data

Target Type Estimate

Performance Estimate

Algorithm(ATR)Sensors

Network Resources databases,communications, etc.

System Performance Analysis

ResourceAllocation

Active Computations Concept

Successively refined inferences Time or resources to achieve performance goal. Additional data required to achieve performance goal.

9


Total System PerformanceResource Consumption Approximations

• ATR system performance entails more than just accuracy:– Time to classify a target– Electrical power dissipation– Sensor engagement, CPU cycles, bits communicated, and other

“opportunity costs”

• Need real-time estimates of total system performance– Enable informed tradeoff of ATR accuracy with throughput and network

resource consumption– Dynamically adapt the system as requirements, capabilities, and

operational scenarios evolve

12


Active Computation• The need to actively manage computations is

acute in complex, time-critical environments– Information has a time value– Some information now may be better than a lot of information later,

after it is too late to take decisive action– Ideally, we’d like some information now and more later

• Static ATR implementations perform the same computations for every image they receive– No tentative answers are available before processing is finished– Availability of more time will not improve the solution accuracy

13


Active Computation• Dynamic ATR systems employ active

computations to maximize the time-value (or resource-value) of information

• Approach: Generate a sequence of increasingly accurate classifications– More resources are consumed at every stage– Continue until accuracy is good enough, resource cap is reached,

or the result is no longer relevant– Control the computations to maximize the total information value

14


• Maximum-likelihood ATR solution is

• Solve a sequence of simpler problems– The functions get closer to with each stage– Each problem is easy to solve given previous solutions– Let be the sequence of problems that are chosen up to stage k– Let be the error and be the resources used– The best strategy at stage K minimizes the total expected cost

Active Computation

15


• Seek heuristic strategies that do not require prior knowledge of K, but are nearly optimal for all K

• For example, maximize the expected future increase in likelihood

Active Computation

16



• Motivation: – ATR; Network Centric Warfare; Biometrics;

Image Understanding• Active Computations• Achievable Rate Regions


17


Recognition System Capacity:More Motivation

● Number of bits for recognition● Number of patterns that can be distinguished● Number of bits to extract from data● Size of long term memory● Data and processing dependence● Start with simple i.i.d. model

18


X1

X2

.

.

.XMc

Selectone

p(x) X1

X2

.

.

.XMc

p(y|x)

y

Selectone

p(h)

xh

Objective: Pr{g(V)=h}>1-ε, s.t. R=(Rc, Rx, Ry)

U1,U2,…,UMx

fmemoryencoder

gh

φsensoryencoder

V

memoryrepresentation

sensoryrepresentation

Rc

Rx Ry

19


Pattern Recognition Codes and Achievable Rates

p(x)

x1 x2 x3

xMc

… p(y|x)

(i)i

v(i)

11…11My

…00…1000…0100...00…321

…

xh yselect patternp(h)

g2

…

y

f g1

(i)i

u(i)

11…11Mx

…00…1000…0100...00…321

…

p(x)

x1 x2 x3

xMc

… p(y|x)

(i)i

v(i)

11…11My

…00…1000…0100...00…321

…

(i)i

v(i)

11…11My

…00…1000…0100...00…321

…


select patternp(h)

g2

………

y

f g1

(i)i

u(i)

11…11Mx

…00…1000…0100...00…321

…

(i)i

u(i)

11…11Mx

…00…1000…0100...00…321

…

(i)i

u(i)

11…11Mx

…00…1000…0100...00…321

…

p(x)

x1 x2 x3

xMc

… p(y|x)

(i)i

v(i)

11…11My

…00…1000…0100...00…321

…


…

y

f

(i)i

u(i)

11…11Mx

…00…1000…0100...00…321

…

p(x)

x1 x2 x3

xMc

… p(y|x)

(i)i

v(i)

11…11My

…00…1000…0100...00…321

…

(i)i

v(i)

11…11My

…00…1000…0100...00…321

…


select patternp(h)

………

y

f

(i)i

u(i)

11…11Mx

…00…1000…0100...00…321

…

(i)i

u(i)

11…11Mx

…00…1000…0100...00…321

…

(i)i

u(i)

11…11Mx

…00…1000…0100...00…321

…

p(x)

x1 x2 x3

xMc

… p(y|x)

(i)i

v(i)

11…11My

…00…1000…0100...00…321

…


g2

…

y

g1

(i)i

u(i)

11…11Mx

…00…1000…0100...00…321

…

p(x)

x1 x2 x3

xMc

… p(y|x)

(i)i

v(i)

11…11My

…00…1000…0100...00…321

…

(i)i

v(i)

11…11My

…00…1000…0100...00…321

…


select patternp(h)

g2

………

y

g1

(i)i

u(i)

11…11Mx

…00…1000…0100...00…321

…

(i)i

u(i)

11…11Mx

…00…1000…0100...00…321

…

(i)i

u(i)

11…11Mx

…00…1000…0100...00…321

…

p(x)

x1 x2 x3

xMc

… p(y|x)

(i)i

v(i)

11…11My

…00…1000…0100...00…321

…


…

y

(i)i

u(i)

11…11Mx

…00…1000…0100...00…321

…

p(x)

x1 x2 x3

xMc

… p(y|x)

(i)i

v(i)

11…11My

…00…1000…0100...00…321

…

(i)i

v(i)

11…11My

…00…1000…0100...00…321

…


select patternp(h)

………

y

(i)i

u(i)

11…11Mx

…00…1000…0100...00…321

…

(i)i

u(i)

11…11Mx

…00…1000…0100...00…321

…

(i)i

u(i)

11…11Mx

…00…1000…0100...00…321

…

20

O’Sullivan AS-MURI 2006Characterizing

21


0

I(X;U)

I(Y;V)

H(Y)

H(X)

0

Rx > I(X;U)Ry > I(Y;V)Rc < I(U;V)-I(U;V|X,Y)

On the border,

U-X-Y-V , so

R*=R**=R.

V=Y

U=X

Rc=I(X;Y)

`Unlimited’ U,V capacity: U=X, Y=V

Rc < I(X;Y)Random channel coding

Rc=0

Rc=0

Poor memory: U=0

Rc < I(0;V)=0

Poor senses:V=0

Rc<I(U;0)=0.

Rc=I(X;Y)-I(X;Y|U)

Rc=I(X;Y)-I(X;Y|V)

22


f

g

φ

(U,V)

X

Yp(x,y)

A Related Gap: the distributed source coding problem

- Problem: Characterize the achievable (Rx,Ry,Dx,Dy)-Sergio Servetto claimed solution at ITW 2006-Solution should transfer to our problem

23


Related Work

25


Sensor Data Link

Weapon LinkShooter Link

•Recce Imagery (SAR, IR, Visible)•Intelligent Bandwidth Compression

Ground Recce/Intel Station Strike Planning System

•Wide Area Cueing (ATC)

•Select Target

Targeting Info Link•Link Target data to TOC(type, location, motion,…)

•Strike Planning•Weapon Selection

•ATR•Reference Library, Aimpoint

Terminal ATR

•Strike plan•Target location•ATR parameters

Naval Impact/Payoff: The Sensor to Shooter Problem

26


Convex Hull Inner BoundWestover and O’Sullivan ISIT 2005

• The convex hull of this inner bound is achievable.• Consider the set of all distributions such that conditioned

on a random variable Q, we have U – X – Y – V. Then the achievable region is

• For every case examined, this convex hull is achieved by time sharing between a length 4 MC and the (0,0,0) point.

)|;(

)|;()|;(

QVUIR

QVYIRQUXIR

c

y

x

),,( 1111 cyx RRRR

}2,,2,1{}2,,2,1{:

}2,,2,1{:

}2,,2,1{:

11

1

1

1

1

1

cy

y

x

nRnR

nRn

nRn

g

Y

Xf

),,( 2222 cyx RRRR

}2,,2,1{}2,,2,1{

}2,,2,1{}2,,2,1{:

}2,,2,1{:

}2,,2,1{:

)(

)(2

)(2

)(2

121

121

12

12

ccc

yyy

yy

xx

RRnnR

RRnnR

RRnn

RRnn

g

Y

Xf

Coarse Code: (f1,Φ1,g1)n Refining Code: (f2,Φ2,g2)n

The rate sextuplet (Rx1,Ry1,Rc1,Rx2,Ry2,Rc2) is achievable if there exist sequences of recognition codes (f1,Φ1,g1)n and (f2,Φ2,g2)n such that Comment: two different systems (different patterns)

.0, 21 ne

ne PP

Successive Refinement, Two-Stage RecognitionGiven a sequence of (Mx1, My1, Mc1, n) pattern recognition codes, design a sequence of (Mx2, My2, Mc2, n) PR codes with the first sequence as subcodesMx1≤ Mx2 My1 ≤ My2 Mc1 < Mc2

“Up and to the right”

Achievability: Inner Bound

);(

);();(

111

11

11

VUIR

VYIRUXIR

c

y

x

),;,(

),;(),;(

21212

212

212

VVUUIR

VVYIRUUXIR

c

y

x

Theorem: Two-stage recognition is achievable if there existauxiliary random variables U1, V1, U2, and V2 satisfying• Markov conditions: U1 – X – Y – V1 and

(U1,U2) – X – Y – (V1,V2).• Rate Bounds:

Inner Bound: Proof Sketch. At the coarse stage:•Use the coding strategy for the single-stage pattern recognition systemAt the refining stage:•Given memory and sensory indices from the coarse stage, generate “refining” codebooks according to the conditional distributions p(u2|u1(·)) and p(v2|v1(·)).•Encode memory and sensory data with pairs of indices corresponding to coarse and refining stage•Use typical-set decoding to identify pattern

29


Inner Bound: Successive Refinement

);(

);();(

111

11

11

VUIR

VYIRUXIR

c

y

x

);(

);();(

222

22

22

VUIR

VYIRUXIR

c

y

x

Analogous to the Markov condition for successive refinement for rate-distortion.

Equitz and Cover, “Successive refinement of information,” IEEE Trans. Info. Theory, Mar. 1991.

Corollary: Successive refinement is achievable if there existauxiliary random variables U1, V1, U2, and V2 satisfying• Markov condition: U1 – U2 – X – Y – V2 – V1. • Rate bounds:

30


Converse: Outer BoundTheorem: If the rate sextuplet (Rx1,Ry1,Rc1,Rx2,Ry2,Rc2) isachievable then there exist auxiliary random variables U1,V1, U2, and V2 satisfying• Markov conditions: U1 – X – Y and X – Y – V1 and

(U1,U2) – X – Y and X – Y – (V1,V2). • Rate Bounds:

),|;();(

);();(

11

111

11

11

YXVUIVUIR

VYIRUXIR

c

y

x

),|,;,(),;,(

),;(),;(

2121

21212

212

212

YXVVUUIVVUUIR

VVYIRUUXIR

c

y

x

Corollary: Two length 4 Markov chains follow: U1 – U2 – X – Y and X – Y – V2 – V1

31


Extension: Hierarchical Recognition Based on Random Labels

• Extend results so that the codebook is the same for the two stages

• Randomly label each Xn(k) with a label L(k), out of exp[n(Rc2-Rc1)] labels

• For each label, use a (Mx1, My1, Mc1, n) pattern recognition code

• Given Yn, run every decoder (for every label) list of exp[n(Rc2-Rc1)] possible patterns

• Use refinement codebooks to determine label and therefore the pattern

32


Extension: Hierarchical Recognition Based on Hierarchical Pattern Model

• Assume that the patterns are generated by a hierarchical model W X (class identity)

• Inner Bound: U1 – W – Y – V1 and U1 – U2 – X – Y – V2 – V1

• Use a (Mx1, My1, Mc1, n) pattern recognition code to obtain Wn(i) (class)

• Use refinement codebooks to determine Xn(i,j) (identity)

))(|( i.i.d. 2 ..., ,2 ,1 ),,(

)( i.i.d. 2 ..., ,2 ,1 ),()( 12

1

iwxpjjiX

wpiiW

kRRnn

nRn

cc

c

p(w)

)1,1(nX

)1(nW

)2(nW

)2( 1cnRnW

)2,1(nX

)2,1( )( 12 cc RRnnX

)1,2(nX)2,2(nX

)2,2( )( 12 cc RRnnX

)1,2( 1cnRnX)2,2( 1cnRnX

)2,2( )( 121 ccc RRnnRnX

p(x|w)

p(x|w)

p(x|w)

33


Extensions• Inner bounds for prototypical examples:

Gaussian, binary. Convex hull is achievable by successive refinement.

• Successive refinement “up and to the right”

34


Recognition System Design• Developing robust ATR algorithms, deriving

limits on recognition performance• Quantifying recognition performance as a

function of system resource measures• Developing algorithms and implementations

that adapt to dynamically varying resource constraints

– Time, availability of processors, communication bandwidth, data storage, sensor image quality

• Impact: increase efficiency and effectiveness of system implementations

– Information latency problem– Recognition systems using visual

imagery, SAR, ladar– Increase in ATR performance– Allow more imagery to be screened– Provide systematic tools for analyzing

design choices such as processors and network communication

Collaborators Washington UniversityJoseph A. O’SullivanAndrew LiNaveen SinglaPo-Hsiang LaiLee Montagnino Brandon WestoverRobert PlessRonald S. IndeckNatalia A. Schmid (UWVa)Michael D. DeVore (UVa)Alan Van Nevel

35


36


Selected Limitations of Existing Systems

• “Stovepipe” design– Fixed inputs, processing, database, output– Fixed time– Algorithms are not transparent

Seek “any-time” adaptive system design

37


Naval Capability Provided“ Network centric warfare is military

operations that exploit information and networking technology to integrate widely dispersed human decision makers, situational and targeting sensors, and forces and weapons into a highly adaptive, comprehensive system to achieve unprecedented mission effectiveness.” Network-Centric Naval Forces, Naval Studies Board, National Research Council, 2000

Active Computations Exploit technology Integrate sensors, resource allocation,decision makers, algorithms Adapt to dynamically varying resources Provide measures of uncertainty as aa function of available resources

Recognition System Capacity

Documents