A Prescriptive Adaptive Test Framework (PATFrame) for Unmanned and Autonomous Systems: A Collaboration Between MIT, USC, UT Arlington and Softstar Systems.

A Prescriptive Adaptive Test Framework (PATFrame) for Unmanned and

Autonomous Systems: A Collaboration Between MIT, USC, UT Arlington and Softstar Systems

Dr. Ricardo ValerdiMassachusetts Institute of Technology

March 10 ,2010

http://lean.mit.edu © 2010 Massachusetts Institute of Technology Valerdi- 2http://lean.mit.edu

Outline

• The Challenge

• PATFrame team

• PATFrame objectives & scope

• PATFrame features

• Next steps

“Anything that gives us newknowledge gives us an opportunity to be more rational”

- Herb Simon

Sponsors

Transition Partners

http://lean.mit.edu © 2010 Massachusetts Institute of Technology Valerdi- 4http://lean.mit.edu

PATFrame kickoff meeting Fort Hood, TX - Aug 2009

http://lean.mit.edu © 2010 Massachusetts Institute of Technology Valerdi- 5http://lean.mit.edu

The Challenge: Science Fiction to Reality

Singer, P. W., Wired For War: The Robotics Revolution and Conflict in the 21st Century (Penguin, 2009)

“You will be trying to apply international law written for the Second World War to Star Trek technology.”

http://lean.mit.edu © 2010 Massachusetts Institute of Technology Valerdi- 6http://lean.mit.edu

PATFrame Objective

To provide a decision support tool encompassing a prescriptive and adaptive framework for UAS SoS Testing• PATFrame will be implemented using a software dashboard that

will enable improved decision making for the UAS T&E community

• Focused on addressing BAA topics TTE-6 Prescribed System of Systems Environments and MA-6 Adaptive Architectural Frameworks

• Three University team (MIT-USC-UTA) draws from experts in test & evaluation, decision theory, systems engineering, software architectures, robotics and modeling

http://mit.edu/patframe

http://lean.mit.edu © 2010 Massachusetts Institute of Technology Valerdi- 7http://lean.mit.edu

Prescriptive Adaptive

Test Strategy/Test Infrastructure

System under test

Prescriptive Adaptive Test Framework

http://lean.mit.edu © 2010 Massachusetts Institute of Technology Valerdi- 8http://lean.mit.edu

Hours Years

Long Term

Planning

Decisions/

investments

Data Collection,

analysis &

reprioritization

Test

Planning

(in SoS

environment)

Time Scale for Testing Decisions

Days

PATFrame Scope

Test

Development

(i.e., design for

testability)

Months

Test Execution

(real time)

Minutes

http://lean.mit.edu © 2010 Massachusetts Institute of Technology Valerdi- 9http://lean.mit.edu

S SoS

Complexity of the systemunder test

Autonomy

of sys

tem

under test

AI

Human

SoS

none

Net-centricity of the environment

Diffi

culty

Testing a SoS in SoS environment

Testing SoS

Testing a system in a SoS environment

net-centric focus

UAST focus

Ultimate GoalDARPA Urban Grand Challenge Use case(UAS in SoS)

PATFrame Initial Focus:Testing AutonomousSystem in SoS environment

http://lean.mit.edu © 2010 Massachusetts Institute of Technology Valerdi- 10http://lean.mit.edu

Normative

Descriptive

Prescriptive

Goal: Capture Actual SoS Test

Goal: Construct Theoretical Best “SoS” Test

Metric set, “best” levels

Metrics, state of the practice levels

Goal: Synthetic framework for SoS testing at single and multi-program level

“Successful SoS Test” = f(metricA, metricB, etc.)

Actual SoS tests include metricA’, metricC, etc.

limit(MetricA)

Normative (best)

Descriptive (SoP)

Descriptive (actual)

Potential (new test A)

test A contribution to state of the practice

“success”

MetricA MetricB MetricC MetricN

…

…

Prescribed System of Systems Environment

http://lean.mit.edu © 2010 Massachusetts Institute of Technology Valerdi- 11http://lean.mit.edu

Normative

Descriptive

PrescriptiveDecision Support System

Real OptionsSystems Dynamics

Leading Indicators

...

PATFrame Decision Support Technologies

PATFrame Core Models

PATFrame Adaptive Architecture Framework

Assessment of Effectiveness

Testbed

Ontology

Modeling Language for Adaptive Test System Architectures

Modeling, Analysis, Simulation, and Synthesis Toolsuite

Risk ModelsParametric

Models

Integrated Test Management

http://lean.mit.edu © 2010 Massachusetts Institute of Technology Valerdi- 12http://lean.mit.edu

Real Options as Prescriptive and Adaptive Framework for SoS Testing

• Use case:• Question: what to test? (i.e. what SoS test scenarios to prescribe?)

• Inputs: models of autonomous/net-centric system of systems, major uncertainties

• Outputs: enablers and types of real options for responding to uncertainties as candidate test scenarios (i.e. identification of how SoS can adapt)

Joint/ SoS Model: coupled dependency structure matrix

Identification of Real Options:

Vehicle1

(Army)

Vehicle2

(Navy)

Vehicle3

(Air Force)

Ontology

Uncertainties

Mission objectives

Objective: Maintain Vehicle1Vehicle2 comm.Uncertainty: proximity of vehicles 1 and 2

1. Real option to adjust comm. range using flexible range comm. systems on vehicles1, 22. Real option to use Vehicle3 as relay

1

2

3

http://lean.mit.edu © 2010 Massachusetts Institute of Technology Valerdi- 13http://lean.mit.edu

Testing to Reduce SoS Risks vs. the Risks of Performing Tests on an SoS

http://lean.mit.edu © 2010 Massachusetts Institute of Technology Valerdi- 14http://lean.mit.edu

14

Example PATFrame Tool Concept

•Question:• When am I Done testing?

• Technology:• Defect estimation model

• Trade Quality for Delivery Schedule

• Inputs:• Defects discovered

•Outputs:• Defects remaining, cost to quit, cost to continue

http://lean.mit.edu © 2010 Massachusetts Institute of Technology Valerdi- 15http://lean.mit.edu15

When am I done testing?

Knowledge Base

Analysis Techniques

Risk & Uncertainty Analysis(leading indicators)

Process Analysis(System Dynamics)

Reasoning Engine

Plan

Analyze and Design

Execute

Evaluate

•What realizable options are available?

•Which test do I runand in what order?

Primary Outputs for Test and Evaluation

•When am I done•Testing?

Test Requirements

Primary Inputs to PATFrame

Models of the system under test

Available resources •and time

Mission needs/goals

Model of the SoS environment

Ontology

•How should my test •strategy change?

PATFrame