Rank Robustness of Composite Indices: Dominance and Ambiguity

Rank Robustness of Composite Indices:

Dominance and Ambiguity

James E. Foster George Washington University & OxfordMark McGillivray Ausaid Suman Seth Vanderbilt University

Composite IndicesMany multidimensional indices take the form

C(x;w) = w·xwhere

x = (x1,x2,…,xD) is a vector of dimensional achievements and

w = (w1,w2,…,wD) is a vector of weights satisfying

w1,…,wD 0 and w1+···+wD = 1

“Composite index”

Human Development Index or HDI (UNDP) Multidimensional Povery Index Mα (Alkire/Foster) Gender Empowerment Index (UNDP) Global Peace Index (Inst. Econ Peace) Environmental Sustainability Index (Yale) Child Well-being Index (Inst. Child Dev.) US News College Ranking

Why this Form? Natural Easy to understand Statistical properties

Key challenge…how to choose weights?

Examples

Methods Normative Statistical e.g. Principal component analysis Equal weights

“Our argument for equal index weights is based on the premise that no objective mechanism exists to determine the relative importance of the different aspects of environmental sustainability.”

Environmental Sustainability Index “…we have no reliable basis for doing [otherwise]”

Mayer and Jencks, 1989

Note: Analogous to “Principle of insufficient reason”

Choice of Weights

Choice of WeightsNote

Inevitable arbitrariness in choice of weightsArrow’s critique of HDI’s weights w = (1/3,1/3,1/3)

Why important?Comparison could be ambiguous or robust

C(x;w0) > C(y;w0)at initial w0 and yet

C(x;w) < C(y;w)at some other reasonable w

“Ambiguous”

Example: Ambiguous Comparison

w = (1/2, 1/4, 1/4)w0 = (1/3, 1/3, 1/3)

Country HDIIreland (x) 0.956

Canada (y) 0.950

Country HDIIreland (x) 0.937

Canada (y) 0.942

C(x;w0) > C(y;w0)at initial w0 and

C(x;w) > C(y;w)at all reasonable w

“Robust”

Example: Robust ComparisonCountry HDIAustralia (x) 0.957

Sweden (y) 0.951

Same ranking for all w

How to discern?Clearly,

Ranking C(x;w0) > C(y;w0) reveals nothing about the robustness or ambiguity of the comparison

The following look the same – but are different

Goal: Provide intuitive criteria for distinguishing between robust and ambiguous comparisons

Country HDI

Ireland (x) 0.956

Canada (y) 0.950

Country HDI

Australia (x) 0.957

Sweden (y) 0.951

Our ApproachDefinitions

D Simplex of dimension D

w0 Initial weighting vectorW Set of “reasonable” weighting vectors around

w0

W is non-empty

(1,0,0)

(0,0,1) (0,1,0)

w0

Our Approach

Notation For any a, b in D, a b denotes ad bd for all d; a > b denotes ad bd with a ≠ b for all d; and a >> b denotes ad > bd for all d

X D: non-empty set of alternatives to be ranked

Define weak robustness relation RW on X by x RW y if and only if C(x;w) C(y;w) for all w W

Characterize RW among all binary relation R on X Based on Bewley (1986)

Our ApproachCharacterization of RW Axioms of R

Quasiordering (Q): R is transitive and reflexive Monotonicity (M): (i) If x > y then x R y; (ii) if x >>

y then y R x cannot hold. Independence (I): Let x, y, z, y', z' X where y' =

x +(1–)y and z' = x + (1–)y' for 0 < < 1. Then y R z if and only if y' R z'.

Continuity (C): The sets {x X | x R z} and {x X | z R x} are closed for all z X.

Our ApproachCharacterization of RW Theorem 1 Suppose that X is closed, convex, and

has a nonempty interior. Then a binary relation R on X satisfies axioms Q, M, I, and C if and only if there exist a non-empty, closed, and convex set W D such that R = RW

Interesting interpretation Maximin criterion of Gilboa and Schmeidler (1989) for multiple priors

Our Approach

Robustness of rankingx CW y if and only if C(x;w0) > C(y;w0) and x RW y

Analogous constructs Knightian uncertainty (Bewley, 1986)Partial comparability (Sen, 1970)Poverty ordering (Foster-Shorrocks, 1988)

Q/ Which W?A/ We use nested sets De where e [0,1]

Epsilon-Contamination model of ambiguity (Ellsburg, 1961)

The C0 Relation

Suppose W = {w0} = D0 Denote resulting relation by C0 so that

x C0 y if and only if C(x;w0) > C(y;w0)Original complete ordering of the composite index

Interpretation of D0

Supremely confident in choice of initial weightings - offers no robustness test at all

Implausible

HDI Top Ten According to C0

Rank Country HDI1 Norway 0.965

2 Iceland 0.960

3 Australia 0.957

4 Ireland 0.956

5 Sweden 0.951

6 Canada 0.950

7 Japan 0.949

8 United States 0.948

9 Switzerland 0.947

10 Netherlands 0.947

Human Development Report 2006, UNDP

Coun. Nor Ice Aus Ire Swe Can Jap USA Swit NethRank 1 2 3 4 5 6 7 8 9 10

Nor 1Ice 2 C0

Aus 3 C0 C0

Ire 4 C0 C0 C0

Swe 5 C0 C0 C0 C0

Can 6 C0 C0 C0 C0 C0

Jap 7 C0 C0 C0 C0 C0 C0

USA 8 C0 C0 C0 C0 C0 C0 C0

Swit 9 C0 C0 C0 C0 C0 C0 C0 C0

Neth 10 C0 C0 C0 C0 C0 C0 C0 C0 C0

Complete Ordering

Column Dominates Row

The C1 Relation

Suppose W = S = D1 Denote resulting ranking by C1 so that

x C1 y if and only if C(x;w0) > C(y;w0) and x R1 y

where x R1 y denotes C(x;w) C(y;w) for all w S

Interpretation of D1

No confidence in choice of initial weightings – offers full robustness test

Stringent requirement

Characterization of C1

e1 = (1,0,0)

e2 = (0,1,0) (0,0,1) = e3

Simplex D

Let C(x;ei) = xi; i = 1,2,3

w0

Theorem 2 Let x, y X. Then (i) x R1 y if and only if x ≥ y and (ii) x C1 y if and only if x > y

Example: Robust Comparison

e3 = (0,0,1)

e1 = (1,0,0)

e2 = (0,1,0)

Swe x3 : 0.949Swe x1 : 0.922

Swe x2 : 0.982

Swe HDI : 0.951

Aus x1 : 0.925Aus HDI : 0.957

Aus x3 : 0.954

Aus x2 : 0.993

x C1 y if and only if x > y

Aus C1 Swe

Coun. HDI x1 x2 x3

Aus 0.957 0.925 0.993 0.954

Swe 0.951 0.922 0.982 0.949

w0

Example: Ambiguous Comparison

e3 = (0,0,1)

e1 = (1,0,0)

e2 = (0,1,0)

Can x3 : 0.959

Can x1 : 0.919

Can x2 : 0.970

Can HDI : 0.950

Ire x1 : 0.882 Ire HDI : 0.956Ire x3 : 0.995

Ire x2 : 0.990

Coun. HDI x1 x2 x3

Ire 0.956 0.882 0.990 0.995

Can 0.950 0.919 0.970 0.959

Ireland Canada ranking is not fully robust.

e1 = (1,0,0)

e2 = (0,1,0) (0,0,1) = e3

Can

Ire

w0 w0

Recall C0 Ranking

Column Dominates Row

Coun. Nor Ice Aus Ire Swe Can Jap USA Swit Neth

Rank 1 2 3 4 5 6 7 8 9 10

Nor 1

Ice 2 C0

Aus 3 C0 C0

Ire 4 C0 C0 C0

Swe 5 C0 C0 C0 C0

Can 6 C0 C0 C0 C0 C0

Jap 7 C0 C0 C0 C0 C0 C0

USA 8 C0 C0 C0 C0 C0 C0 C0

Swit 9 C0 C0 C0 C0 C0 C0 C0 C0

Neth 10 C0 C0 C0 C0 C0 C0 C0 C0 C0

Fully Robust Ranking C1

Column Dominates Row

Coun. Nor Ice Aus Ire Swe Can Jap USA Swit Neth

Rank 1 2 3 4 5 6 7 8 9 10

Nor 1

Ice 2

Aus 3

Ire 4

Swe 5 C1

Can 6 C1

Jap 7

USA 8

Swit 9 C1

Neth 10 C1

Partial Ordering Ce

Consider any e satisfying 0 e 1Define

De = {w' S : w' = (1 – e)w0 + ew for some w S}

Interpretation1 – e degree of confidence in initial weighting w0

Ellsburg (1961)

e “size” of resulting set for checking robustnessEx

D1 = D lowest degree of confidence in w0, largest setD0 = {w0} highest degree of confidence, smallest setDe = intermediate

Partial Ordering Ce

e1

De

(1-e)w0 + ee1 = ve1

w0

1-e

e

Partial Ordering Ce

Suppose W = De Denote resulting ranking by Ce so that

x Ce y if and only if C(x;w0) > C(y;w0) and x Re ywhere x Re y denotes C(x;w) C(y;w) for all w De

w0De

e1

e3e2

Characterization of Ce

Denoteve

d = (1- e)w0 + eed

xed = ve

dx Value of x at ved

xe = (xe1, …,xe

D) Vector of these values

w0De

ve1

e1

e3e2

ve2

ve3

Characterization of Ce

Theorem 3 Let x, y X. Then (i) x Re y if and only if xe ≥ ye and (ii) x Ce y if and only if xe > ye

w0De

ve1

e1

e3e2

ve2

ve3

Partial Ordering Ce

e3 = (0,0,1)

e1 = (1,0,0)

e2 = (0,1,0)

Recall Canada/Ireland example

Can

Ire

Recall Fully Robust Ranking C1

Column Dominates Row

Coun. Nor Ice Aus Ire Swe Can Jap USA Swit Neth

Rank 1 2 3 4 5 6 7 8 9 10

Nor 1

Ice 2

Aus 3

Ire 4

Swe 5 C1

Can 6 C1

Jap 7

USA 8

Swit 9 C1

Neth 10 C1

Ce for e = 1/4

Coun. Nor Ice Aus Ire Swe Can Jap USA Swit Neth

Rank 1 2 3 4 5 6 7 8 9 10

Nor 1

Ice 2

Aus 3 Ce

Ire 4 Ce

Swe 5 Ce Ce Ce

Can 6 Ce Ce Ce

Jap 7 Ce Ce

USA 8 Ce Ce Ce

Swit 9 Ce Ce Ce

Neth 10 Ce Ce Ce Ce Ce

Column Dominates Row

Measure of Robustness

IdeaHow robust is a given comparison?

DenoteA = C(x,w0) - C(y,w0) difference in HDI’s

B = maxwS{C(y,w) - C(x,w)} max dimensional departure

Definer = A/(A+B) Measure of robustness

Measure of Robustness

Theorem 4 Suppose that x C0 y for x, y X and let r be the robustness level associated with this comparison. Then the e-robustness relation x Ce y holds if and only if e ≤ r.

Measure of Robustness

Size of the largest sub-simplex is the measuure of robustness r

e1 = (1,0,0)

e2 = (0,1,0)

Can

Ire

e3 = (0,0,1)

Robustness Calculation

Coun. HDI x1 x2 x3

Ire 0.956 0.882 0.990 0.995

Can 0.950 0.919 0.970 0.959

Difference 0.006 -0.037 0.02 0.036

A = 0.006B = 0.037r = 0.006/(0.006 + 0.037) = 0.139

Robustness Calculation

Coun. HDI x1 x2 x3

Aus 0.957 0.925 0.993 0.954Swe 0.951 0.922 0.982 0.949Difference 0.006 0.003 0.011 0.005

A = 0.006 B = - 0.003 r = 1

Note that both comparisons have same A

Yet very different robustness levels!

Coun. Nor Ice Aus Ire Swe Can Jap USA Swit NethRank 1 2 3 4 5 6 7 8 9 10

Nor 1Ice 2 20 Aus 3 35 19 Ire 4 86 14 4 Swe 5 53 94 100 11 Can 6 61 100 60 14 14 Jap 7 28 34 23 9 7 2 USA 8 77 28 17 67 5 3 1 Swit 9 49 100 41 16 17 20 6 2 Neth 10 100 68 57 47 25 13 4 7 1

Measure of Robustness (%)

Column Dominates Row

Conclusion

Propose a tool for measuring robustness of ranking The idea is motivated by partial orderings, epsilon-

contamination, and Knightian uncertainty The tool can be extended to the case of general

means (e.g., HPI) A measure of robusteness is proposed Robustness Vs. Redundancy (McGillivray 1991)

Thank you