0 0 Exploring a Range of Decision Making Techniques Nov 18,2009 John Bates ( [email protected] ) [email protected].

1

1

Exploring a Range of Decision Making Techniques

Nov 18,2009

John Bates ([email protected])

22

The Range of Decision Making Techniques

Considerations when selecting a technique…

How important is the Decision; who will review decision? How much data/information do you have or can you get? How much time and $ do you have?

Discussion

Group (BOGSAT

)

Poorly Done

Decision Analysis

Decision Analysis

Incorporating Mod & Sim

Decision Analysis

Incorporating SMEs

Coin Tossor

Guess

Least Desirable - Most DesirableAdvantages/Disadvantages to each technique.

Fielded system;

record data throughout lifecycle.

OperationalExercise and

Testing

33

Symptoms of poorly done decision analysis

Lack of creative and significant Alternatives

Solving the Wrong Problem/Objective

Not involving the real decision makers

Competing Objectives

Avoiding Uncertainty

Success measured on weight and thickness of final brief

Not sure about the definition of the attributes

Provided the solution at the beginning

44

How important is your decision?

55

Recommend a structured process that includes all stakeholders and is traceable/defendable

• Scoring scales can be nominal, ordinal, interval, or ratios

• Utility curves transform scoring scales into a common measurable output (Utility)

• Weighting of the tree structure determines relative importance

• There are various methods to determine importance

• AHP

• Relative Weighting

• Overall Goal: Determine best solution to fill need

• Sub-Goals:

• Maximize Technical Performance, Measures of Effectiveness

• Maximize Operational Suitability

• Minimize Risks, etc.

• Criteria must be

independent, discriminatory, meaningful, and measurable

• Assess each alternative relative to the criteria in the decision tree

• Utility curves then transform the scoring scales into measurable data ranging from 0 to 1 with 1 being the most preferred

• Rank ordering of alternatives can be further decomposed to determine major contributors to the overall decision

• Conduct estimates for each alternative

• Can use cost objectives and thresholds

• Cost estimates and be treated separate or added into the evaluation model

• Sensitivity analysis can be conducted to analyze the results due to:

• Changes in weights

• Changes in scores

• Utility scores can be can be graphed versus cost to determine the "Best Value" alternative

• Determine Requirements

• Identify/Clarify Goal

• Define Criteria

1 • Determine Priorities

• Develop Utility Curves

• Develop Scoring Scales

2• Conduct

Sensitivity Analysis

• Determine “Best Value”

5

• Assess Alternatives

• Rank Order Alternatives

4• Generate Alternatives

• Develop Estimates

• Incorporate Estimates into Model

3

Understand Problem/Vision Modeling Decide and Communicate

66

Prioritization (Weighting) TechniquesAnalytic Hierarchy Process

Speed Reliability Length Range sum row avgSpeed 1 3 1 3 0.375 0.705 0.107 0.188 1.375 0.344

Reliability 1/3 1 7 9 0.125 0.235 0.750 0.563 1.673 0.418Length 1 1/7 1 3 0.375 0.034 0.107 0.188 0.703 0.176Range 1/3 1/9 1/3 1 0.125 0.026 0.036 0.063 0.249 0.062

2.667 4.254 9.333 16.000 1.000 1.000 1.000 1.000 4.000 1.000

Weightings To

Prioritize User

NeedsCalculation:

1/ 2.667= 0.375Sum Rows Calculate Row Average

(Called the Row Average of the

Normalized Columns (RANC) method)

Assign Scores Using a Scale

Sum Column

Developed by Thomas L. Saaty in the 1970s. The AHP provides a comprehensive and rational framework for structuring a decision problem, for representing and quantifying its elements, for relating those elements to overall goals, and for evaluating alternative solutions.

• Determine Priorities

• Develop Utility Curves

• Develop Scoring Scales

2

Importance

1 Equal

3 Moderate

5 Strong

7 Very Strong

9 Extreme

77

Capturing SME input

Participants will be asked to evaluate criteria through a series of pairwise comparisons, which calculates each factor’s relative importance.

To capture these comparisons, each participant uses a remote control to assign a number (1-9) that reflects the relative weight or importance of each decision factor.

Criteria #1 Criteria #2

1 Intensity of Importance

2 3 4 5 6 87 99 8 7 6 5 34 2

Speed Reliability Length Range sum row avgSpeed 1 3 1 3 0.375 0.705 0.107 0.188 1.375 0.344

Reliability 1/3 1 7 9 0.125 0.235 0.750 0.563 1.673 0.418Length 1 1/7 1 3 0.375 0.034 0.107 0.188 0.703 0.176Range 1/3 1/9 1/3 1 0.125 0.026 0.036 0.063 0.249 0.062

2.667 4.254 9.333 16.000 1.000 1.000 1.000 1.000 4.000 1.000

• Determine Priorities

• Develop Utility Curves

• Develop Scoring Scales

2

# Inputs = (N * N-1)/2

88

Methodology for collecting group input

Pairwise Comparisons ensure all combinations are considered independently

The Analytic Hierarchy Process is used to determine the criteria

priority. Pairwise comparisons are made for each unique combination.

During the session, the facilitator will focus on areas of disagreement by identifying divergent points of view and encouraging participants to elaborate.

Participants are encouraged to state their opinions and change their minds – and votes – if they so desire. During facilitated discussion, participants may continue to disagree on the relative importance of different decision factors but they will have a better understanding of different views within the group.

• Determine Priorities

• Develop Utility Curves

• Develop Scoring Scales

2

99

Decision Analysis at multiple levels?

Mission Areas

UIC (ASSIGNED TO RESPORG)

Capability

Area 1Capability

Area 2

Capability Area 3

PESTO Resource Elements

Capability 2 Capability 3Capability 1

O

T

S

E

P

NTA2NTA1

Joint Capability Areas

(Future)

NUMBER OF INPUTS

REQUIRED

~10

~50

~250

~3000

• Determine Priorities

• Develop Utility Curves

• Develop Scoring Scales

2

1010

Relative Scoring Example

Solution Mission A Formula % Importance

Car 1 =1/4 25%

Tank

Plane 3 =3/4 75%

Boat

TOTAL 4 100%

Criteria Scale

Supports Mission Success 1

Important to Mission Success 3

Critical to Mission Success 9

• Determine Priorities

• Develop Utility Curves

• Develop Scoring Scales

2

1111

Expert Choice – Great discussion/consensus using clickers, takes a lot of time

Group Systems – Great at capturing multiple user input simultaneously , can be performed remotely (surveys), requires extensive setup.

Excel AHP – Used to Prioritize Resource Categories within a level - Simple to setup, easy to facilitate potential for a lot of intersections ((n * n-1)/2), lose input variation and correlation checking

Excel (0,1,3,9) Scale – Simple, quick, needs to be strongly facilitated, lose input variation and consistency check

The choice of techniques is primarily based on number of intersections and time needed to perform the analysis

TIME TO COLLECT

Summary of Different Prioritization Techniques

1212

Scaling Examples for Alternative Scoring

Utility

Signature Improvement in undetectable range (meters)

1

0

0 1000

Utility

Range (meters to Refuel)

1

0

2142 6000

The range for the Measure of Merit for Signature is based on the improvement in detection range the vehicle realizes through IR signature reduction.

The range of the Measure of Merit for Range is based on the ORD Requirement on meters to recognition.

• Determine Priorities

• Develop Utility Curves

• Develop Scoring Scales

2

1313

Alternative Comparison Technique

DIV A

DIV B

DIV C

Capability/MOE Definition: The ability to transport passengers

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 ….

T

O

T

O

T

O

Number of Passengers

• Determine Priorities

• Develop Utility Curves

• Develop Scoring Scales

2

1414

Functional Analysis Conduct Operations

AND AND

Command and Control (C2)

Survive Threats and Conditions

Surveillance (Find/Fix/Track/ID/Assess)

Launch RecoverEmbark Team/Equip

Transit to

OpArea

Transit from

OpAreaInterdict

Escort Vessel

Force Protection

Logistics &Service Support

Plan/DirectOrganize

Train/Equip Arm/ Fuel

C2

Employment (Air)Employment (Sea)

Employment (Fwd Basing)

Maneuverability/Navigation

Employment (Sea)

Employment (Fwd Basing)

Speed

Seakeeping

OR Abort Mission

Patrol

Speed

Deploy

Waterborne Checkpoint, PSHD, HAD, Vessel Escort

Enabling Tasks (Off-board)

Enabling Functions (Off-board)

Key Capabilities/MOPs

LEGEND

Capacity

1

3 4 59

11

8

10

12

13Detectability

OR

Speed

Lethality

Sustain Force15

14

Endurance

Loiter

7

16

Staging

Survivability

6

Transportability

2

Systems Engineering-based Functional Decomposition provides structured process to define Required Mission Capabilities to assess Alternatives

Systems Engineering-based Functional Decomposition provides structured process to define Required Mission Capabilities to assess Alternatives

1515

Modeling and Simulation Techniques

Discrete Event Simulation

Behavior/Functional Modeling

Physical or Prototype Modeling

Wargaming/Simulations

Sensitivity Analysis

Linear Programming

Reliability ,Failure Modes and Effects Analysis

Mathematical/Monte Carlo or Probability Modeling

1616

Method to Communicate Analysis

Results of Analysis

1. System B

2. System A

3. System C

1717

Method to Communicate Analysis

TRL LEVELS

9-10

7-8

<6

LCCE vs Effectiveness for Ten Craft with the Least Number of Threshold Failures

50' Riverine Patrol Craft (CB-90)

47' Concept Riverine Craft

40' Riverine M-Hull Craft

40' Riverine Patrol Boat

40' Riverine Craft (SURC)

37' Catamaran HVA Escort

36' Unmanned RHIB

36' Navy STD RHIB (Cabin)

31' Riverine Assault Boat

31' Riverine Craft (SOCR)

$6,000,000

$9,000,000

$12,000,000

$15,000,000

$18,000,000

3.60 3.70 3.80 3.90 4.00 4.10 4.20 4.30 4.40 4.50Effectiveness

LCCE (FY07$)

1818

Another Method of Communicating Decisions

This quad chart represents the capability areas that should be pursued first.

Not Addressed

Communications Gear

782 Gear

Ancillary Equipment

Ammunition

M49 Telescope Scope SASRScope Sniper Rifle

SASRSniper Rifle

0

4.5

9

-5 -4 -3 -2 -1 0 1 2 3 4 5

Quad 1High PriorityPoor Capabilities

Quad 3Low PriorityPoor Capabilities

Quad 4Low PriorityGood Capabilities

Quad 2High PriorityGood Capabilities

1919

The Range of Decision Making Techniques

So what are the key considerations when selecting a technique…

How important is the Decision; who will review decision? How much data/information do you have or can you get? How much time and $ do you have?

2020

Questions

?