1
Analytical Support for Rapid Initial Assessment
Charles Twardy, Ed Wright, Kathryn Laskey,
Tod Levitt, Kellen Leister, Andy Loerch
George Mason University C4I Center
2
Topics • Rapid Initiative Assessment (IA) challenges
• Overview of Mason’s IA methodology
• Example
2
3
Challenge: Analysis Support for Initial IA We focus here:
rapid initial assessment.
Models can also be reused here and beyond.
3
4
• Rigorous, rapid, consistent, re-usable analytic justification for JIEDDO initiative assessments
• Fulfills critical need as warfighter requirements grow while budgets tighten and scrutiny increases
Solution: Analysis Support for Initial IA
Rapid Initial IA Requirements • Provide rapid assessments (days to
weeks) • Model dependence of relevant Measures
of Effectiveness (MOEs) on system & environmental variables
• Use available knowledge • Identify information collection priorities • Be consistent, repeatable, & extensible
5/19/10
1. _ 2. _
3. Implement as Probabilistic Model
4. Exercise Model & Analyze Results
5. Determine Sensitive Parameters
6. Report Results
Initiative Assessment Process
6
1. Identify MOEs
2. Generate Explanation
Example MOEs: Casualties per Incident Time to Complete Mission Weapons Intelligence Gathered
Partial Explanation Example: If there is an IED detonation during robot neutralization, Blue soldiers are not exposed. The robot may be damaged or destroyed.
Bayesian network (BN) model for
EOD robot
7
IA Approach Benefits • Consistent framework for
assessing initiatives.
• Clearly communicates to decision makers, the assessed impact, potential tradeoffs, and the mechanism by which it works.
• Makes the explanation structured, explicit, executable, and reusable.
• Perform what-if, try scenarios, test understanding, perform sensitivity analysis.
• Enable development of more informative test plans.
• Identify relevant MOEs.
• Generate an Explanation of how the initiative is expected to affect MOEs.
• Implement the explanation as a probabilistic model.
• Execute & analyze model to assess performance
• Determine the “sensitive parameters” (SPs) to help prioritize information collection.
ExplanationProbabilistic Model • Generate explanation of how initiative affects MOE
– Clutter can interfere with the ability of the sensor to detect IEDs and cause false positives
• Implement explanation as Bayesian network (BN) – Structured, explicit, executable, and reusable – Models how initiative is likely to perform in
operation – Supports what-if and sensitivity analysis
5/19/108
CPT for Sensor_Result
9
Available Knowledge • SMEs (at JIEDDO and elsewhere) • JUONS and other needs statements • Initiative documentation • Current suite of equipment & capabilities • Additional contractor knowledge • Blue and Red TTPs • Previous initiatives • Previous models • Previous tests
9
New Class
?
New Initiative
Select model From
Repository
No
Model Analysis
Add
Performance Assessment &
Sensitive Parameters
OFFLINE: Enhance models
Yes
Model Sufficient?
Yes
Add
No (n+1)th
Iteration Model
nth Iteration Model
Model Development Spiral
Recursive Spiral Prototyping
10
Time Avail
?
No
Yes
Develop 1st Iteration Model
Modify Model
Model Repository
Analysis
5/19/1011
Try Scenarios in the Model, and examine the effect on the MOEs
For each MOE, find the most influential variables:
Intel. Potential redDetonatesRobot
redDetonation probDisableSuccess robotProbEffective
robotReadiness
Calculate individual link strengths:
Vary some parameters over their range:
12
Specific Technologies RECCE I
Cougar 6x6 Platform
Gyrocam (VOSS)
Remote Wpn Sys
EOD Robot
Comms
Duke v1
RECCE II Adds
LNS
Duke (v2)
EOD Robot In the Remote Deployment System
Remote Wpn Sys
VOSS on Mast
LNS
Photo from the (S) ATEC C&L Report, July 2008
Example 1 EOD Robot
Assess EOD Robot
- New Class? Yes
Select MOEs
Build 1st Iteration Model
13
New Class
?
New Initiative
Select model From
Repository
No
Model Analysis
Add
Performance Assessment and
Sensitive Parameters
OFFLINE: Enhance models
Yes
Model Sufficient?
Yes
Add
No (n+1)th
Iteration Model
nth Iteration Model
Recursive Spiral Prototyping
Time Avail?
No
Yes
Develop 1st Iteration Model
Modify Model
Model Repository
New Initiative
New Class
?
Yes
Develop 1st Iteration Model
MOEs by Tenet *
14
Tenet Potential MOEs Predict (Intell. Gathered)… Mission
Time (Cost)
Prevent Number and relative proportion of each type of IED tactic, Number or percentage of interceptions, raids, captures before emplacement, #IEDs/mission mile
Detect-Air P(detect), False Alarm Rate, Sweep Width, Rate of Advance Detect-Ground
P(detect), False Alarm Rate, Sweep Width, Rate of Advance, P(spot)
Neutralize P(neutralize), Neutralize Time, Intelligence Gathered Mitigate Casualties/Attack, KIA/Attack, WIA/Attack, Damage/Attack
Some MOEs suggested by Perry et al., Minimizing the Threat from Improvised Explosive Devices in Iraq, RAND 2007 Some from the RECCE II Initiative Evaluation Plan (AMSAA, August 2008)
*Tenet: JIEDDO divided initiatives into “tenets” which roughly follow the “left of boom” timeline.
15
Identification of MOEs Assumptions The EOD robot provides a capability to remotely neutralize (disable or detonate) an IED. If the robot is not available or not successful, a soldier will neutralize the IED.
MOE Assumptions and Considerations Time Robot may take longer than an EOD soldier
If the robot is unsuccessful, we still must use a soldier P(neutralize by robot)
Distinguish disable from destroy
Casualties or Damage per Attack
• Replace with generalized, qualitative P(damage) • If Red detonates the IED during robot neutralization, soldiers are not
exposed. The robot may be damaged or lost. • If the robot is unavailable, or fails, then a soldier will be at risk. • If the IED is not spotted, robot has no effect on damage / casualties.
P(collecting valuable Intelligence)
• If Blue disables the IED, it can be examined for forensic intelligence. • If Blue detonates it, there may be some intelligence collected before
the detonation. • If Red detonates it, there is little intelligence gained.
16
Implement Explanation as BN Model assumes IED is present and successfully detected. • If robot is available and working
correctly, it can be used to attempt to disable or detonate an IED.
• If the robot succeeds in disabling the IED, we can gather forensic intel.
• Little intelligence can be collected if the robot detonates the IED.
• If there is a Red detonation during neutralization, Blue soldiers are not exposed. The robot may be damaged or destroyed.
• If the robot is not available or not successful, a soldier will be at risk while disabling the IED.
• Using the robot may take longer than using an EOD soldier.
• If unsuccessful, a soldier must still disable the IED.
1
2
2
3
3
1
Example 1 EOD Robot (2)
Assess EOD Robot
- New Class? Yes
Build 1st Iteration Model Add 1st Iteration Model to Repository
Time Available? No
Run the Model, Analysis
17
New Class
?
New Initiative
Select model From
Repository
No
Model Analysis
Add
Performance Assessment and
Sensitive Parameters
OFFLINE: Enhance models
Yes
Model Sufficient?
Yes
Add
No (n+1)th
Iteration Model
nth Iteration Model
Recursive Spiral Prototyping
Time Avail?
No
Yes
Develop 1st Iteration Model
Modify Model
Model Repository
New Initiative
New Class
?
Yes
Develop 1st Iteration Model
Select model From
Repository
Add
nth Iteration Model
Model Repository
Model Sufficient?
No Time Avail?
No
Model Analysis
Performance Assessment and
Sensitive Parameters
Robot Analysis 1: View Effects If the robot is not available … a soldier will be at risk while disabling the IED.
If a robot is available and it is working correctly, it can be used to attempt to remotely disable or detonate an IED.
Lower risk to soldier, more time
If the robot succeeds in disabling the IED, it can be examined for forensic intelligence. Less intelligence can be collected if the robot detonates the IED.
18
19
Robot Analysis 2: Sensitive Parameters by MOE EOD Robot: Top 5 Sensitive Parameters by MOE.
• Assuming robotAvailable, and excluding deterministic functions
ClearTime Intelligence Damage redDetonatesRobot redDetonatesRobot redDetonation
redDetonation redDetonation redDetonatesRobot robotReadiness probDisableSuccess robotProbEffective
robotProbEffective robotProbEffective robotReadiness probDisableSuccess robotReadiness --
Next Steps (as time allows):
• Investigate sensitive parameters in more detail
• Extend / refine the model: additional variables, situations; extend or refine the state space of important variables; refine local probability distributions.
• Identify knowledge requirements for the sensitive parameters
• Seek additional information: SMEs, system documents, data collection, …
20
Summary Challenge or Need IA Methodology Solutions
Short timeline Spiral development: start simple, extend later Reuse models; Standard flow starting from MOEs
Little quantiative data / need to assess prior to testing
Probabilistic models can use available expert and prior knowledge as soft constraints; initiative models make use of any existing models onto which they are added
Need analytical support Probabilistic models are explicit representations of how the initiative is thought to work, and can be executed.
Prioritize information collection
Sensitivity analysis in the model can rank variables by influence, and show the effect of parameter changes
Consistent, repeatable, extensible
Standardized methodology based on MOEs leads to consistent assessment across initiatives. Model reuse provides repeatability and extensibility
Integrate with Portfolio Management
MOE statistics from the model feed into PM approach: casualties/damage, time, effectiveness