Defence Research andDevelopment Canada
Recherche et développementpour la défense Canada Canada
Identification of Data Sets for a Robustness Analysis
Micheline Bélanger 1, Jean-Marc Martel 2, Adel Guitouni 1
1RDDC Valcartier, 2Université Laval
10th ICCRTS
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Agenda
• Canadian Forces Operations Planning Process (CFOPP)
• Multicriteria Decision Aid (MCDA) Methodology
• Robustness Analysis
• Identification of possible data sets based on DM local preferences
• Identification of plausible data sets
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
InitiatingDirective
SOR
Orientation ConceptDevelopment
Political&
MilitaryAssessment
PlanDevelopment
CPG CONOPs OPLAN
Canadian OPP and Estimate Process
PlanReviewInitiation
Mission AnalysisMission Statement
COAsDevelopment
COAsRefinement
COAs War gaming
COAsComparison
Information Brief
Decision Brief
Estimate Process
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Comparison of COAs
COA2
COAm
COA1
...
COA1
COA2
COAm
...
Ranking or SelectionSet of COAs
Decision Maker’sPreferences
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Initial Decision Maker’s Preference: COAsEvaluation Criteria
Flexibility
Complexity
SustainabilityOptimum Use of Resources
Factors
Risk
Covering Operational Tasks
Covering Mission’s Locations
Covering Enemy’s CoAs
Operation Complexity
Logistic Complexity
C&C Complexity
Sustainability
Cost of Resources
Criteria
Impact of Sensors Coverage Gaps
Military Personnel Loss
Collateral Damages
Equipment Reliability
Personnel Effectiveness
Confrontation Risk
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Comparison of COAs
COA2
COAm
COA1
...
Decision Maker’sPreferences
COA1
COA2
COAm
...
Best possible compromise considering:–Conflicting evaluation criteria–DM’s values and preferences
Set of COAs Ranking or Selection
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Comparison of COAs
COA2
COAm
COA1
...
Decision Maker’sPreferences
COA1
COA2
COAm
...
MCDAMethodology
Set of COAs Ranking or Selection
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Applying MCDA Methodology to Compare COAs
DM ’sPreferences
Criteria (1…n)C1 ... Cj ... Cn
a1 e11 ... e1j ... e1n
: : : : : :ai ei1 ... eij ... ein
: : : : : :CoA
s (1…
m)
am em1 ... emj ... emn
CoA1
CoA2
CoAm
...
MulticriterionAggregationProcedure
Local Preferences
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Modelling Decision Making Styles: Introduction of Local Preferences Modelling
• Each criterion is assigned a coefficient of relative importance (πj), which might represents
– a “trade-off” or a “voting power”
• When comparing two COAs, three types of thresholds are introduced
– Indifference (qj) thresholds
– Preference (pj) thresholds
– Veto (vj) thresholds
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Modelling Decision Making Styles: Introduction of Local Preferences Modelling (2)
• Indifference (qj) thresholds represent:
– the highest difference between the evaluations of two COAs, according to a given criterion j, for which the decision-maker is incapable to make a clear choice between these two alternatives, given that everything is the same otherwise
• Preference (pj) thresholds represent:
– the smallest difference between the evaluations of two alternatives, according to a given criterion j, for which the decision-maker is able to make a clear choice of one, given that everything is the same otherwise
• Veto (vj) thresholds represent:
– the smallest difference between the evaluations of two alternatives, according to a given criterion j, for which the decision-maker cannot conclude that an alternative ai is as good as ak, if the performance of akis higher than the performance of ai and if the difference of the evaluations between them is greater than νj (even if the performance of ai is higher than ak for all others criteria).
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Possible Preference Relationship When Comparing two COAs
ki aa ~k i a a φ ik aa φ
j ij p e − jij pe + kje ije jij q e + jij qe −
k i φ fa a ikφfaa
jj
jkjijki
jkjijjkfji
jkjijki
qpwhere
qeeaa
peeqaa
peeaa
>
≤−⇔
<−<⇔
+≥⇔
j
j
~
φ
φ
indifference between ai and ak
strict preferenceof ai over ak
strict preferenceof ak over ai
weak preferenceof ai over ak
weak preferenceof ak over ai
DiscriminationThresholds
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Problematic
• Modelling requires transforming, reduction and decomposition of the reality
– It is impossible to derive exact models of the situation
• The complexity of the military operation context prevents from deriving exact and precise values to represent Commanders preferences structures (command style)
• Very high likelihood for more than one plausible data set to represent the Decision Maker’s preferences structure
– Possibility to get more that one “optimal”solution for the same decision-making situation
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Why a Robustness Analysis
• The imperfection of the data set obtained should be properly considered in decision analysis
• Robustness analysis should consider all plausible data sets in order to identify a robust ranking of plausible good decisions (COAs)
• A specific set of data instantiates a potential realization of the model of decision.
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Robustness [Kouvelis and Yu, 1997]
• From the point of view of the optimality
– The solution of a mathematical program is qualified as robust, if it remains in neighbourhood of the optimum for all plausible data sets of the model
• Generalisation from optimality to best compromise.
• Since the approach of robustness is crucially based on the process of generation of plausible data sets, it requires a good knowledge of the environment in which the decision take place
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Robust ranking approach proposed
• Three critical steps were identified for robustness analysis in the context of COAs comparison:
– an approach to model all the data sets that instantiate the decision-maker’s preferences, which are “not so well known”;
– a method to aggregate the pre-orders generated from each data set;
– a robustness criterion suited for the decision-making situation.
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Identification of Possible Data Sets
PlausibleCoefficients of
relative importance of the Criteria
PlausiblePreference Modelling
ThresholdsFiltering Plausible
Data Sets
Plausible Data Sets
Possible Data Sets
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Coefficients of relative importance of the criteria (CRIC)
• Identification of intervals
– based on decision-maker’s intervals
– based on decision-maker’s explicit values
[π1(j) ,π2
(j)] njj ,...,1,10 =<< π
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
CRIC Based on decision-maker’s explicit values
( ) ( ) ( )nj ggg ,...,,...,1is the least important criterionis the most important criterion ( )1g
( )ng
[ ] [ ] [ ]2)(
1)(
2)(
1)(
2)1(
1)1( ,,,,,,, nnjj ππππππ ΚΚ
1,...,2and,1,0with 1)1(
2)(
2)1(
1)(
2)1(
1)( −=∀≤≥⟨⟩ −+ njjjjjn ππππππ
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
CRIC Based on decision-maker’s explicit values (2)
( )( ) ( )
2
21jj
j
πππ
+=Normalized with:
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Ex oequo Intervals
( )
( ) ( )
( )
( ) ( ) ( )
( )
( ) ( )[ ]
( ) ( )[ ] ( ) ( )[ ]
( ) ( )[ ]
( ) ( )[ ] ( ) ( )[ ] ( ) ( )[ ]
( ) ( )[ ]25
15
24
14
24
14
24
14
23
13
22
12
22
12
21
11
5
444
3
22
1
,
,,,,,
,
,,,
,
,,
),(
ππ
ππππππ
ππ
ππππ
ππ
g
ggg
g
gg
g
[ ] [ ] [ ]2)(
1)(
2)(
1)(
2)1(
1)1( ,,,,,,, nnjj ππππππ ΚΚ 1and0with 2
)1(1
)( ⟨⟩ ππ n
Process ex oequoas one (block)
( ) ( ) ( )nj ggg ,...,,...,1is the least important criterionis the most important criterion ( )1g
( )ng
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
CRIC Based on a Pre-Order of Importance
π(1) ≥ π(2) ≥ …. ≥ π(n) with πj > 0 and , ∑=
=n
jj
1
1π
Example with 15 criteria :
Consider to have 6 data setsπ1 → the c.r.i. are equally balanced π6 → the c.r.i. are decreasing from 1 to 1/n
Reduce the values by slices on a basis of
(n-2)/4
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Indifference Thresholds
– DM interval– Otherwise
• DM value
• Default value
→ 80% and 60%
[ ] 0with,, 121 ≥∀ jjj qjqq
jj EXq 25.015.0' =
'jq
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Preference Thresholds
– DM interval– Otherwise
• DM value
• Default value
→ 80% and 60%
[ ] jpp jj ∀21 , jjjj Epqp ≤≥ 221 and
( )jjjj qvEp −+= 05.025.0'
'jp
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Veto Thresholds
– DM interval
– Otherwise
• DM value
• Default value
→ 80% and 60%
[ ] 2121 with, jjjjj pvvvv >⇒
j
jj
Ev
π
25.0=
jv
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Filtering Plausible Data Sets
• To reduce the number of data sets
– For each interval, use only 3 data
• First value, middle one, last one
– Treatment of parameters as groups or blocks
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Discussion
• Characteristics of the proposed military decision-making model – A=(a1,…,ai,…,am);– Λ/C=(g1,…,gj,…,gn);– E=(eij =gj(ai), i=1,...,m; j=1,....,n);– M=(πj, vj(eij),qj(eij),pj(eij), i=1,...,m; j=1,...,n);
and– a multicriterion method, PAMSSEM, within the
framework of the ranking problematic.given m alternatives and n attributes/criteria
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Discussion
• Identification of possible values for:
– coefficient of relative importance (πj)
– discrimination thresholds
• indifference (qj )
• preference (pj )
– veto thresholds (vj )
• Identification of plausible data sets
Defence R&D Canada – Valcartier • R & D pour la défense Canada – Valcartier
Conclusion
• Robust results should be less influenced by the imperfection of the data occurring in the evaluations of the courses of actions as well as in the instantiation of parameters representing decision-maker’s preferences during the modeling process of a military situation
• Considering all plausible information that might represent the decision-making context– Not constrained to a single data set
• Robustness concept should be generalised to other information/knowledge analysis methodologies– e.g.; IPB and Enemy’s estimates