Toward mission-specific service utility estimation using analytic stochastic process models Dave Thornley International Technology Alliance .

Toward mission-specific service utility estimation using analytic

stochastic process models Dave Thornley

International Technology Alliancehttp://usukita.org

Imperial College London

Quality, Utility, Value

• Quality of Information (QoI) used as a focus for comprehension, generality and communication– What does it mean?– What else could have told me this?– What guarantees can we provide?– Supports choices during action

• Utility of information (UoI) or another service supports design choices– Will this system support our achievement of goals, and how well?– Will it still work when we’ve finished with it– Can we sell it or its information products?

• Given utility estimates for some purpose, we can assess the value that should/will be ascribed (VoI)– Will be ascribed “This piece of information makes my life easier.”– Should be ascribed “No it doesn’t”

Mission Abstraction, Requirements and Structure

• PLANs provide structure and projections

• PHYsics includes sensor models, traffic generators and environmental modulators

• INTelligence includes receipt of signals, fusion, storage, hypothesis and dissemination

• Situational Awareness maps knowledge to awareness and understanding (more next slide)

• Decision Maker is a representation of the human in the loop

• ACTion maps decisions to physical outcomes via effectiveness measures

PLAN

PHY

INT*

SA*

DM*

ACT+

+outcome probabilities*confusion matrix

Abstract stochastic perspective

• In a given deployment, predictable outcomes are influenced by sensing service design choices

• A sensing package that results in better outcomes for the same plan is providing higher quality of information amortized over the mission, and is of higher utility specifically to that mission

• Consider the outcomes as a locus of possibilities, which may be a combination of discrete and continuous variables (target location and assessment, sensor energy remaining and functional integrity).

• A stochastic model associates probabilities with states as a function of time. If we ensure that there is a state defined for each outcome we care about, we can quantifies the contributions of alternative services in characteristics that can be meaningfully compared

Information driven model

• Detection cues tracking• Tracking enables focus of various types

through intelligence gathering• Alternative competing hypotheses are

evaluated using intelligence product arrivals and retrieval/ refactoring to achieve focus and situational awareness

• Decisions drive action or instruct sources• Action creates feedback

Keithley’s Knowledge Matrix Quality Requirement

Location Track Identity Activity Capability Intent

5 5m Vectors & prediction

Specify object and hierarchy

Many actions, states and linkages

Many factors & influence

Desired end state and intent for future ops known

4 10m Vectors Specify object

Many actions and states: several linkages

Several factors & influence

Desired end state known and intent for future ops determined

3 20m General speed and direction

Specify type Several actions and states: one linkage

Few factors & influence

Desired end state and intent for future ops determined

2 100m Toward or away

Distinguish object

Few actions and states: no linkages

Few factors & no influence

Desired end state determined and intent for future ops inferable

1 1km Stationary or not

Discriminate object

Single action or state

One factor & no influence

Desired end state inferable and intent for future ops inferable

0 10km Detect Detect Detect Detect Desired end state inferable and intent for future ops unknown

The matrix was originally developed to assess the value of information fusion algorithms to C4ISTAR missions to justify the cost of their development.

ISR requirements are specified in terms of a canonical set of questions. The questions need to be supported by details of the required QoI for the mission to succeed.

The ISR question is answered at the level of the commander’s information requirements not the data level.

BUT insufficiently flexible to allow a detailed consideration for matching resources to dynamic mission requirements.

Timed stochastic outcome modeling for utility comparison

A B A+

B-B

A B+

A-

Earlier More predictable Safer (abort before negative effects begin)

System A has higher utility than B according to performance prediction

Methods

• Performance Analysis Process Algebra– Compositional timed stochastic modeling– Abstract to information product delivery and

operational modes– Can be massaged into a range of solution tools– Native model is a continuous time discrete state

Markov chain• Equilibrium solutions

– Measurement of consumption and exposure• Transient solutions

– Response time predictions– Evolution of accuracy achievable

INCIDER

• DSTL human factors team• Our example scenario is lifted directly and simplified

somewhat from one of their presentations:

www.dodccrp.org/events/2006_CCRTS/html/presentations/025.pdf

Also see:Dean, D., Vincent, A., Mistry, B., Hossain, A., Spaans, M. and

Petiet, P., “Representing a Combat ID Analysis Tool within an Agent Based Constructive Simulation”, The International C2 Journal, Vol 2, No 2

Isolated decision making scenario

EO

ScoutMidScout

Near

Mid

Far

Near

DM

Obstructiveterrain

Decisionpoint

Maximum EO detection range

Maximum EO recognition range

TID reception limit

TID

Scenario components

FAT <signals> ( (Sensors <evidence> SA) <policy> DM )

• Policy includes orders and tests• Signals include EO interpretation, TID comms, Scout

vision and HQ picture• Evidence raises, lowers or sets confidence in Red and

Blue hypotheses• Model that can generate Red and Blue traffic, and the

SA maintenance and decision making sequences for each has 1597 states, with a 5 phase Erlang FAT transition process

MARS Federated Analytic Traffic

• Entities are modeled as states that combine, in our example– Location – space subdivide according to invariants in

the response of the mission– Class, affiliation &c. (mood?) – just Red/Blue here

• Multiple sensing modalities must be modeled and correlated, so traffic centralized, and formed of components, each representing an entity or group of entities

FAT traffic progress

0 5 10 15 20 25 30 35 40 45 500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

time from reaching detection range

prob

abili

tyProbability of occupying ranges

Far

MidNear

Decide

MARS Situational Awareness

• Confidence in each hypothesis Red, Blue

• Example has zero, low, medium, high– In general, these demarcations will be

selected according to regions on the real line that do not change the outcome of fusion

• Predicates calculated on these states– Comparison of values (less/greater/equal)

Evolution of SA and decisions

0 5 10 15 20 25 30 35 40 45 500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

time from reaching detection range

prob

abili

tyProgression through detection and ID of Blue (Bold = no scout)

Correct Pass

Prefer RedPrefer Blue

Fratricide

Decision QoI

5 10 15 20 25 30 35 40 45 50

0.95

0.96

0.97

0.98

0.99

1

Dec

isio

n Q

oI

5 10 15 20 25 30 35 40 45 50

0.01

0.02

0.03

0.04

0.05

Dec

isio

n ra

te

Time from detection at Far

Instantaneous decision QoI and rate modeled from detection in Far

Decision QoI

Decision rate

Decision making utility

05 10 15 20 25 30 35 40 45 50

0

0.01

0.02

0.03

0.04

0.05

0.06

Time from initial detection

Rat

e o

f dec

isio

n m

akin

g

Pass and fratricide rates without scout, and with scout stationed at base or at observation point

Frat no scout

Pass no scout

Frat base scout

Pass base scout

Frat dedicated scout

Pass dedicated scout

05 10 15 20 25 30 35 40 45 50

0

0.01

0.02

0.03

0.04

0.05

0.06

Time from initial detection

Rat

e of

dec

isio

n m

akin

g

Pass and fratricide rates without scout, and with scout stationed at base or at observation point

Frat no scout

Pass no scout

Frat base scout

Pass base scout

Frat dedicated scout

Pass dedicated scout

Mission abstraction• Priors on encounters and

conditions enable definition of a traffic and environment generator

• Intelligence services formulated and composed

• Situational awareness maintained by an abstraction of the fusion functions to map intelligence products to SA upgrades

• Decisions taken by recognizing SA patterns

• Actions pursued leading to feedback to the mission physics

PLAN

PHY

INT*

SA*

DM*

ACT+

+outcome probabilities*confusion matrix

Abstracting space

• QoI emission characteristics constrain asset selections and operational modes

• Regions of validity of service output can be defined

• Optimization requires amortization over mission projections

y

x

3

5

8

2

694

1

7

Track accuracy low in 8, but overall service may satisfy mission ISR

C and A6 or B6

1: CA62: CA6 or CB63: CB64: CA35: A6B66: CB37: A3B68: QoI delivery target not met9: A6B3

Amortizing costs

• States can be defined for a composite sensing service in which measures of interest conform to appropriate invariants– A specific combination of

assets is active– Battery energy

consumption is approximately constant

– Personnel are at definable risk

For example, calculate branching probabilities and transition delay distributions over state space to give a semi-Markov chain.

3

5

8

2

694

1

7

Conclusions=(question,T).Conclusions;

• We have shapes for UoI comparison• We can make those shapes from timed stochastic

process models• Those models can also estimate QoI• The models are a link between QoI and UoI• VoI that is subjective because of a situational awareness

horizon may be found by, for example, marginalizing the dependency structures found in the equilibrium and transient solutions to the models

• When we manage to work in human factors, we’ll have a handle on heuristically subjective VoI

• I hope this was an absorbing talk