Top Banner
Michaela Saisana 5th Impact Assessment Course JRC & SecGen, Brussels, 20-21/01/2015 1 Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana [email protected] European Commission Joint Research Centre Econometrics and Applied Statistics Unit
49

Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Jan 20, 2020

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

1

Impact Assessment Tools:

Multi-criteria Analysis

(the Maximum Likelihood Approach)

Michaela Saisana

[email protected]

European Commission

Joint Research Centre

Econometrics and Applied Statistics Unit

Page 2: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

2

Issues we will touch upon:

• Impact assessment studies

• Cost Benefit Analysis (+ limitations)

• MCA rooted in Social Choice Theory

• 5 methods (Relative majority, Condorcet, Borda, Successive eliminations, Median ranking)

• Weighted Sum (most common; limitations)

• Weights as importance coefficients (BA and AHP)

• MCA: Maximum likelihood approach (steps, suitability)

• Sensitivity Analysis of MCA results

• Conclusion

Page 3: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

3

Impact Assessment Studies • Locating a new plant

• Human resources management

• Evaluating projects

• Selecting an investment strategy

• Electricity production planning

• Regional planning

• Evaluation of urban waste management systems

• Environmental applications

• Health Risk Prediction

• Systemic Risk Assessment ( JRC collaboration with the European Systemic Risk Board)

Page 4: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

4

Page 5: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

5

Basic steps of cost-benefit analysis (CBA)

1. Determine if CBA is worth doing

2. Identify objectives and policy alternatives

3. Determine stakeholders

4. Identify costs and benefits of each alternative

5. Sort into measurable and non-measurable costs and benefits

6. Estimate costs and benefits that can be measured in monetary terms

7. Conduct sensitivity analysis

8. Compare costs-benefits across alternatives

9. Adjust for non-measurable costs and benefits(?)

10. Make a decision

Page 6: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

6

Cost-benefit guidelines

• UK Department of the Treasury, Appraisal and Evaluation in Central Government (The Green

Book), London:2002, http://www.hmtreasury.gov.uk/data_greenbook_index.htm

• NZ Treasury guidelines

www.treasury.govt.nz/publications/guidance/planning/costbenefitanalysis>

• Australian Government, Office of Best Practice Regulation,

http://www.finance.gov.au/obpr/cost-benefit-analysis.html (see especially Handbook of Cost-

Benefit Analysis, and Best Practice Regulation Handbook)

• Queensland Government, Department of Infrastructure and Planning, Cost Benefit Analysis,

www.dip.qld.gov.au/resources/guideline/project-assurance-framework/pafcost-benefit-

analysis.pdf

• Government of Western Australia, Department of Treasury and Finance, 2005, Project

Evaluation Guidelines,

www.dtf.wa.gov.au/cms/uploadedFiles/project_evaluation_guidelines_2002.pdf

Page 7: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

7

Limitations of CBA

• Results often highly sensitive to specific assumptions, such as

discount rate

• Difficult to balance non-quantifiable costs/benefits against

quantifiable ones

• Anthropocentric in its underlying social vision

How much is life, education (literacy), welfare, health, ecological

sustainability, employment (business confidence) worthy?

Page 8: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

8

Multi Criteria Analysis (MCA) - Definition

“Multi Criteria Analysis is a decision-making tool, developed for

complex multi-criteria problems that include quantitative and/or

qualitative aspects of the problem in the decision making process.”

(Center for International Forestry Research, CIFOR, 1999)

Page 9: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

9

MCA - Steps

1. Establish the decision context

2. Identify the criteria and the options

3. Describe/rate the performance of each option against the criteria

4. Assign weights across criteria

5. Combine the information to obtain a ranking of the options

6. Examine the results and review

7. Conduct sensitivity analysis

8. Final decisions

A story next!

Page 10: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

10

Story: particularly insidious form of malaria that devastated Egypt 1942

Cause identified: mosquito

Yet, the real causes were previous expert led interventions:

engineering of railways, irrigation of canals,…

Timothy Mitchell, 2002

Animation by University of California, Berkeley: https://www.youtube.com/watch?v=8jqEj8XUPlk

Page 11: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

11

Tania Murray Li, 2007

These boxes hide:

•Context,

•History (problems are

treated as snapshots)

•Politics (disregard

questions of power and

inequality)

Experts draw themselves

outside the picture

Animation by University of California, Berkeley: https://www.youtube.com/watch?v=8jqEj8XUPlk

Page 12: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

12

MCA - Performance matrix

Criterion 1

(/20)

Criterion 2

(rating)

Criterion 3

(qual.)

Criterion 4

(Y/N) …

Action 1 20 135 G Yes …

Action 2 9 156 B Yes …

Action 3 15 129 VG No …

Action 4 9 146 VB No …

Action 5 7 121 G Yes …

… … … … … …

Criteria should not be dependant on each other and not

redundant (to avoid double counting)

Page 13: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

13

MCA - Performance matrix

• Who decides the ratings?

MCA very flexible wrt who gets a say in either the criteria or rating the

options:

Democratic decision-making - all members of the decision-making body, or each

organizational branch/unit, independently allowed to rate options

Panel of experts asked to make judgments; can use different panel to judge different

criteria

Consensus model - decision-making body ‘thrash it out’

Stakeholder inclusion

Different groups can rate options on different criteria

Page 14: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

14

MCA - Result

The outcome of MCA can be used to:

•Identify a single, most-preferred option

•Rank options

•Short-list a limited number of options for subsequent detailed

appraisal through other methods such as CBA

•Distinguish acceptable from unacceptable options

•Combine different options based on relative strengths

Page 15: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

15

Social Choice Theory Problem:

• A group of voters have to select a candidate among a group of

candidates (election)

• Each voter has a personal ranking of the candidates according to

his/her preferences

•Which candidate must be elected?

What is the «best» voting procedure?

Analogy with multi-criteria analysis:

• Candidates actions

• Voters criteria

Best interest of society

Page 16: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

16

Social Choice Theory

Social choice theory methods would be ideally suited for assessing

multiple options through multiple criteria … and were already available

between the end of the XIII and the XV century, …

Page 17: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

17

1. Ramon Llull (ca. 1232 – ca. 1315) proposed first what would then become known as the

method of Condorcet.

2. Nicolas de Condorcet, (1743 –1794) His „Sketch for a Historical Picture of the Progress of the

Human Spirit (1795)‟ can be considered as an ideological foundation for evidence based policy

(modernity at its best!).

3. Nicholas of Kues (1401 – 1464), also referred to as Nicolaus Cusanus and Nicholas of Cusa

developed what would later be known as the method of Borda.

4. Jean-Charles, chevalier de Borda (1733 – 1799) developed the Borda count.

1 2 3 4

Page 18: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

18

Five methods (among many others)

1. Relative majority

2. Condorcet

3. Borda

4. Successive eliminations

5. Median ranking

Page 19: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

19

3 candidates: Adam, Brian, Carlos

11 voters

10 voters

9 voters

A B C

B C B

C A A

A 11

B 10

C 9

Adam is elected

30 voters:

Method 1 : Relative majority

Page 20: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

20

3 candidates: Adam, Brian, Carlos

11 voters

10 voters

9 voters

A B C

B C B

C A A

A 11

B 10

C 9

Adam is elected

30 voters:

Method 1 : Relative majority

Problem: B and C preferred to

A by a majority of voters!

Page 21: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

21

11 voters

10 voters

9 voters

A B C

B C B

C A A Brian is elected

B preferred to A 19

votes

B preferred to C 21

votes

C preferred to A 19

votes

3 candidates: Adam, Brian, Carlos

30 voters:

Method 2 : Condorcet

Page 22: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

22

4 voters

3 voters

2 voters

A B C

B C A

C A B

A preferred to B 6

votes

B preferred to C 7

votes

C preferred to A 5

votes

Method 2 : Condorcet

3 candidates: Adam, Brian, Carlos

9 voters: Problem: Nobody is elected!

(cycle)

Page 23: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

23 39 x 2 + 31 x 1

Points

2

1

0

30 voters

29 voters

10 voters

10 voters

1 voter

1 voter

A C C B A B

C A B A B C

B B A C C A

Scores

A 101

B 33

C 109

31 x 2 + 39 x 1

11 x 2 + 11 x 1

Carlos is elected!

Method 3 : Borda

3 candidates: Adam, Brian, Carlos

81 voters:

Page 24: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

24

3 voters

2 voters

2 voters

C B A

B A D

A D C

D C B

Points

3

2

1

0

Scores

A 13

B 12

C 11

D 6

Ranking

A

B

C

D

Adam is elected

Method 3 : Borda

4 candidates: Adam, Brian, Carlos, David

7 voters:

Page 25: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

25

3 voters

2 voters

2 voters

C B A

B A C

A C B

Points

2

1

0

Scores

A 6

B 7

C 8

Ranking

C

B

A

Carlos is elected

Method 3 : Borda

4 candidates: Adam, Brian, Carlos, David

7 voters:

Problem: Fully Dependant on

irrelevant alternatives (easy to

manipulate)

Page 26: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

26

Method 4 : Successive eliminations

6 voters

4 voters

1 voters

A C C

C A B

B B A

3 candidates: Adam, Brian, Carlos

11 voters:

Ranking

A

C

B

A tour-wise procedure, whereby

the worst candidate (most voted

in the last position) is eliminated

progressively until one is left.

Page 27: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

27

Method 5 : Median ranking

6 voters

4 voters

1 voters

A C C

C A B

B B A

3 candidates: Adam, Brian, Carlos

11 voters:

A: 11111122223

B:23333333333

C:11111222222

•Ranking of candidates for each

voter

•Median rank for each candidate

across voters

Ranking

A

C

B

Page 28: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

28

5 candidates: Adam, Brian, Carlos, David, Edison

8 voters

7 voters

4 voters

4 voters

2 voters

A B E D C

C D C E E

D C D B D

B E B C B

E A A A A

25 voters: Relative majority Adam elected

Condorcet: Carlos elected

Borda:

David elected

Successive eliminations:

Edison elected

Median ranking:

Carlos elected

?

Page 29: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

29

Page 30: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

30

Kenneth Arrow

(Nobel prize in economy, 1972)

Impossibility theorem (1952):

With at least 2 voters and 3 candidates, it is impossible to build a voting procedure that simultaneously satisfies the 5 following properties:

•Non-dictatorship •Universality • Independence with respect to third parties •Monotonicity •Non-imposition

Page 31: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

31

Most common approach: Weighted sum

Problems: 1) Fully compensatory (elimination of conflicts)

weights

X1

(50%)

X2

(50%)

Y

a 90 10 50

b 10 90 50

c 50 50 50

Page 32: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

32

Most common approach: Weighted sum

Problems: 2) Does not encourage improvement in the weak dimensions

weights

X1

(50%)

X2

(50%)

Y

a 100 10 55

b 20 90 55

c 50 50 50

Page 33: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

33

Most common approach: Weighted sum

Problems: 3) Weights are used as if they were importance coefficients

while they are trade off coefficients

Y = 0.5 ×X1+ 0.5 ×X2

R12 = 0.08, R2

2 = 0.83, corr(X1, X2) =−0.151, V(x1) = 116, V(x2) = 614, V(y) = 162

Page 34: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

34

Most common approach: Weighted sum

• Weighted sum approach only possible under special circumstances (e.g.

standardized variables, uniform covariance matrix…)

• Hence we need to move away from weighted sums …

Effective weights are compared with nominal weights to

ensure coherence between the two.

[Paolo Paruolo, Michaela Saisana, Andrea Saltelli, 2013, Ratings and

rankings: Voodoo or Science?, J. R. Statist. Soc. A, 176 (3), 609-634]

Page 35: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

35

MCA: Maximum likelihood Approach Features (1/2):

• no need for outlier treatment;

• no need for data normalisation;

• no need for uniform covariance matrix;

• no need to attach monetary value to indicator scores;

• no need for data aggregation;

[Kemeny (1959), Young and Levenglick (1978)] Led to: Condorcet-Kemeny-Young-Levenglick (C-K-Y-L) ranking procedure

Page 36: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

36

MCA: Maximum likelihood Approach Features (2/2):

• works with both continuous and categorical variables;

• weights attached to indicators are indeed importance coefficients;

• a compromise between conflicting opinions;

• reasonably resistant to manipulation;

• produces a ranking that is statistically optimal (anonymous, neutral, Pareto optimal, satisfies reinforcement

and local independence of irrelevant alternatives)

[Kemeny (1959), Young and Levenglick (1978)] Led to: Condorcet-Kemeny-Young-Levenglick (C-K-Y-L) ranking procedure

Page 37: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

37

MCA - Performance matrix

Criterion 1

(20%)

Criterion 2

(30%)

Criterion 3

(20%)

Criterion 4

(30%)

Action 1 20 135 G Yes

Action 2 9 156 B Yes

Action 3 15 129 VG No

Action 4 9 146 VB No

Action 5 7 121 G Yes

• Criteria should not be dependant on each other and not redundant (to

avoid double counting)

Where do

weights come

from?

(…next couple

of slides)

Page 38: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

38

43

32

25

29

38

33

38

34

28

39

26

36

39

32

2931

33

36 35

30

34

38

33

29

33 33 33 3335

32

0

10

20

30

40

50

60

70

Co

nsu

mp

tion

Acc

ess

Sta

bili

ty

Nu

triti

on

Qu

alit

y

Qu

alit

y

Ava

ilab

ility

Acc

ess

Sta

tus

Acc

ess

Qu

alit

y

To

ilet

Fa

cilit

ies

Wa

ste

Ma

na

ge

me

nt

Pra

ctic

es

Qu

alit

y

Fa

cilit

ies

En

erg

y

Qu

alit

y

Ava

ilab

ility

Acc

ess

Te

nu

re

Qu

alit

y

Inp

uts

Ski

lls

Se

rvic

es

Ass

ets

Exp

osu

re

Co

pin

g a

bili

ty

Re

cove

ry a

bili

ty

Fo

od

Ed

uca

tion

He

alth

care

Education Farm

Assets

Exposure &

Resilience to

Shocks

Gender

Equality

In 4 dimensions of poverty, the average expert

weight is similar to equal weighting Tiredness

in filling in the questionnaire on weights??

Weights based on Budget

Allocation (42 experts)

[Cohen, Saisana, 2014]

Page 39: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

39

PAIRWISE COMPARISONS;

RELATIVE IMPORTANCE OF ONE CRITERION

OVER ANOTHER 1-9 scale

1 EQUAL 3 MODERATE 5 STRONG 7 VERY STRONG 9 EXTREME

1 2 3 4 5 6 7 8 9

Patents vs. x Royalties x

x Patents vs. Internet x

x Patents vs. Technology exports x

x Patents vs. Telephones x

x Patents vs. Electricity x

Patents vs. x Schooling years x

Patents vs. x University Students x

x Royalties vs. Internet x

Royalties vs. x Technology exports x

x Royalties vs. Telephones x

x Royalties vs. Electricity x

Royalties vs. x Schooling years x

Royalties vs. x University Students x

Internet vs. x Technology exports x

x Internet vs. Telephones x

x Internet vs. Electricity x

Internet vs. x Schooling years x

Internet vs. x University Students x

x Technology exports vs. Telephones x

x Technology exports vs. Electricity x

Technology exports vs. x Schooling years x

Technology exports vs. x University Students x

x Telephones vs. Electricity x

Telephones vs. x Schooling years x

Telephones vs. x University Students x

Electricity vs. x Schooling years x

Electricity vs. x University Students x

x Schooling years vs. University Students x

Which Indicator Do You Feel Is More Important? To What Degree?

Questionnaire

Patents Royalties Internet Tech.Exports Telephones Electricity Schooling University St.

Patents 1 1/3 5 4 3 9 1/6 1/8

Royalties 3 1 3 1/4 5 9 1/3 1/4

Internet 1/5 1/3 1 1/6 2 2 1/7 1/6

Tech.Exports 1/4 4 6 1 5 9 1/4 1/5

Telephones 1/3 1/5 1/2 1/5 1 7 1/9 1/9

Electricity 1/9 1/9 1/2 1/9 1/7 1 1/9 1/9

Schooling 6 3 7 4 9 9 1 2

University St. 8 4 6 5 9 9 1/2 1

solve for the

Eigenvector

Patents 0.109

Royalties 0.103

Internet hosts 0.029

Tech exports 0.117

Telephones 0.030

Electricity 0.014

Schooling 0.301

University st. 0.297

Weights

Inconsistency

17.4 %

Weights based on Analytic Hierarchy Process

[Saisana, Saltelli, 2008]

Page 40: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

40

USING PAIRWISE COMPARISONS, THE

RELATIVE IMPORTANCE

OF ONE CRITERION OVER ANOTHER CAN BE

EXPRESSED

1 EQUAL 3 MODERATE 5 STRONG 7 VERY STRONG 9 EXTREME

Patents Royalties Internet Tech.Exports Telephones Electricity Schooling University St.

Patents 1 1/3 5 4 3 9 1/6 1/8

Royalties 3 1 3 1/4 5 9 1/3 1/4

Internet 1/5 1/3 1 1/6 2 2 1/7 1/6

Tech.Exports 1/4 4 6 1 5 9 1/4 1/5

Telephones 1/3 1/5 1/2 1/5 1 7 1/9 1/9

Electricity 1/9 1/9 1/2 1/9 1/7 1 1/9 1/9

Schooling 6 3 7 4 9 9 1 2

University St. 8 4 6 5 9 9 1/2 1

Weights based on Analytic Hierarchy Process

P=5I

R=3I

We expect:

P > R

Expert said:

R > P (R=3P)

Inconsistency

Page 41: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

41

Performance

matrix

Criterion

1

Criterion

2

Criterion

3

Criterion

4

Criterion

5

Weights 10% 20% 10% 30% 30%

Option A 50 0.6 400 0.6 4000

Option B 70 0.3 500 0.7 5000

Option C 90 0.4 600 0.4 3000

Step 1 - Input matrix to the multicriteria analysis

Example: Three options need to be ranked according to five criteria. The importance

of the criteria is reflected in the respective weights.

MCA: Maximum likelihood Approach

Page 42: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

42

Step 2 – Options are compared pairwise

For each comparison, e.g. option A versus option B, all the weights corresponding to the criteria

that favour A versus B are added up (abbreviated as AB). In this case AB gets the weight of

Criterion 2 only (=0.2). The comparison BA gets the sum of the weights of the remaining

criteria: 1, 3, 4, 5 (=0.8). For n options, there are n (n-1) comparisons to be made. All the values

from the pairwise comparisons are entered in a so called outranking matrix.

Outranking

matrix Option A

Option B

Option C

Option A 0 0.2 0.8

Option B 0.8 0 0.6

Option C 0.2 0.4 0

MCA: Maximum likelihood Approach

Page 43: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

43

Step 3 – Calculate support for all permutations and select the maximum

•All 3! (=6) permutations of the options are considered and the support score for each ranking is calculated.

•ABC has a support of 1.6 (=0.2+0.8+0.6), which is the sum of elements above the diagonal in the

outranking matrix.

•Support scores for all six rankings: ABC= 1.6 |ACB=1.4 | BAC=2.2 | BCA=1.6 | CAB=0.8 | CBA=1.4

•The ranking selected is the one with the maximum likelihood score: BAC

Outranking

matrix Option A

Option B

Option C

Option A 0 0.2 0.8

Option B 0.8 0 0.6

Option C 0.2 0.4 0

MCA: Maximum likelihood Approach

Page 44: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

44

How to shake coupled stairs How coupled stairs are shaken in most of

available literature

MCA: Maximum likelihood Approach

Important to assess sensitivity of results to the weights

Page 45: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

45

Frequency matrix – Sensitivity of the final ranking to the assumptions (e.g.

weights)

Page 46: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

46

• The main limitation of this method is the difficulty in computing the ranking when

the number of options grows (e.g. 50).

• For 10 options 10 = 3,628,800 permutations …still trivial for today’s PCs

• To solve this NP-hard problem when the number of options is very large there are

plenty of numerical algorithms (JRC works on them!)

MCA: Maximum Likelihood Approach

Page 47: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

47

Concluding: How to use MCA in your work

1. Decide on the criteria that you want to use in your assessment;

2. Identify appropriate indicators for each of the criteria (more than one indicator for

each criteria is OK);

3. Score the options on each criterion based on their performance on that criterion;

4. Determine the weights of all the criteria (use for instance AHP);

5. Calculate the overall ranking of the alternatives (e.g., using Maximum Likelihood);

6. Examine the results: explain why some options turn out to be better than others;

7. Do a Sensitivity Analysis: what happens to the results if you change the weights of

the criteria?

8. Make a final decision

Page 48: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

48

[Peyton Young, 1995, Optimal Voting Rules,

Journal of Economic Perspectives 9:51-64]

Peyton Young Professor Emeritus, Research

Professor in Economics,

Johns Hopkins University

The more important issue is whether the (maximum

likelihood) method is intuitively easy to grasp, and

whether it improves on methods currently in use. On both

these counts I think that the answer is affirmative, and

I predict that the time will come when it is considered

a standard tool for political and group decision

making.

Page 49: Impact Assessment Tools: Multi-criteria Analysis · Impact Assessment Tools: Multi-criteria Analysis (the Maximum Likelihood Approach) Michaela Saisana ... Examine the results and

Michaela Saisana

5th Impact Assessment Course

JRC & SecGen, Brussels, 20-21/01/2015

49

http://composite-indicators.jrc.ec.europa.eu

Further reading

MCA •Balinski, M. and R. Laraki (2010). Majority Judgment. Measuring, ranking and electing. MIT Press.

•Balinski, M. and R. Laraki (2014). Judge: Don’t vote!. Operations Research, 62, 483-511.

•Dodgson, J. S., Spackman, M., Pearman, A., and Phillips, L. D. (2009). Multi-criteria analysis: a manual. Department

for Communities and Local Government: London.

•Keeney, R. L. and Raiffa, H. (1976). Decisions with multiple objectives: preferences and value tradeoffs. Wiley, New

York. Reprinted, Cambridge Univ. Press, New York (1993).

•Munda G. (2007), Social multi-criteria evaluation, Springer-Verlag, Heidelberg, New York, Economics Series.

•Young, H. P. (1995). Optimal voting rules. Journal of Economic Perspectives, 9, 51-64.

Weights through expert opinion •Saisana, A. Saltelli, S. (2008) Expert Panel Opinion and Global Sensitivity Analysis for Composite Indicators, Lecture

Notes in Computational Science and Engineering 62, 251-275.

•Cohen, A., Saisana, M., 2014, Quantifying the qualitative: Eliciting expert input to develop the Multidimensional

Poverty Assessment Tool, J of Dev. Studies, 2014, 50(1)).