Advocacy, analysis and quality. The Bermuda triangle of Statistics 59th ISI World Statistics Congress 25-30 August 2013 - Hong Kong Special Administrative Region, China Session STS023 Statistics and policy Andrea Saltelli & Michaela Saisana Joint Research Centre, European Commission
46
Embed
Advocacy, analysis and quality. The Bermuda triangle of ... · Advocacy, analysis and quality. The Bermuda triangle of Statistics 59th ISI World Statistics Congress 25-30 August 2013
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Advocacy, analysis and quality. The Bermuda triangle of Statistics
59th ISI World Statistics Congress
25-30 August 2013 - Hong Kong Special
Administrative Region, China
Session STS023
Statistics and policy
Andrea Saltelli & Michaela Saisana
Joint Research Centre, European
Commission
About indicators
Content
• Statistical indicators for
policy between
modernity and post-
modernity;
• their use for analysis and
advocacy;
• how quality can save the
day;
• when things go wrong
The example of composite
indicators
What official statistics are to the consolidation of the modern
nation state (Hacking, 1990), composite indicators are to the
emergence of post-modernity.
Leibnitz ‘philosophical godfather of Prussian official statistics’ to the Prince Frederik of
Prussia 1700; 56 categories to ‘measure the power of a state’ (the first scoreboard); first
proposal for a statistical office …
Modernity and post modernity; from positivism to constructivism; how ‘Matters of fact’ are
established; Shapin and Shaffer, Latour, …; the emergence of a plurality of norms and views;
the Human Development Index (HDI, 1990) and the explosion of indices…
Composite indicators and post-Modernity
Statistics for policy: three models
A rational-positivist model for the use of indicators and policy
(good quality statistics underpin good policies)
Discursive-interpretive model (statistics contribute to a process
of framing of and focusing on an issue among the many
competing for public's attention)
Strategic model (statistics is used by parties competing for a
given constituency).
see Boulanger, P-M., Political uses of social indicators: overview and application to
sustainable development indicators. International Journal of Sustainable Development,
10 (1,2):14-32, 2007.
Contexts
Composite Indicators
Apples and Oranges
Composite indicators as an object populating a
multidimensional space whose main axes are
advocacy, analysis and quality.
Composite indicators sit between analysis and
advocacy, but quality discriminates the plausible
from the rhetorical.
Advocacy, analysis and quality
These three dimensions (advocacy, analysis and quality) are
not independent from one another.
Most developers adopt for transparency and simplicity linear
aggregation procedures to build composite indicators which
are fraught with considerable difficulties.
In this case quality may suffer at the expenses of advocacy.
Advocacy, analysis and quality
THE ROLE OF COMPOSITE INDICATORS FOR
MEASURING SOCIETAL PROGRESS
Ubiquitous; 6-fold increase in 7 y
Statistics' best known face (to general public & media)
Uncertainty and sensitivity analysis techniques as
tools for the quality assessment of composite
indicators,
J. R. Statist. Soc. A 168(2), 307–323.
Paolo Paruolo, Michaela Saisana,
Andrea Saltelli, (2013),
Ratings and rankings: Voodoo or Science?,
J. R. Statist. Soc. A, 176 (3), 609–634.
Sensitivity analysis
First: The invasive approach
Michaela Saisana, Béatrice d’Hombres,
Andrea Saltelli, Rickety numbers: Volatility of
university rankings and policy implications
Research Policy (2011), 40, 165-177
Sensitivity analysis
Space of alternatives
Including/
excluding variables
Normalisation
Missing data Weights
Aggregation
Country 1
10
20
30
40
50
60
Country 2 Country 3
Sensitivity analysis
Second: The non-invasive approach
Comparing the weights as assigned by developers with
‘effective weights’ derived from sensitivity analysis.
Non invasive Sensitivity analysis
University
Rankings
Comparing the internal coherence of ARWU versus THES (2008) by testing the weights declared by developers with ‘effective’ importance measures. ARWU=Academic ranking of world universities;
THES=Times Higher Education Supplement
declared weight importance
THES X1_Academic opinion: 6354 academics 40% X2_Recruiters’ opinion: 2339 recruiters 10% X3_Full-time equivalent faculty/student ratio 20% X4_Total citation/full time equivalent faculty 20% X5_Percentage of full-time international staff 5% X6_Percentage of full-time international students 5%
Issues with THES: a) ‘Opinion’ variables’ weight overall: >60% instead of 50 b) Faculty/student ratio: 10% instead of 20%
HDI
2009
declared weight importance
Life expectancy, 33%
Adult literacy, 22%
Enrollment education, 11%
GDP per capita, 33%
Non invasive Sensitivity analysis
HDI
2010
Life expectancy, 33%
Education, 33%
GNI per capita, 33%
Non invasive Sensitivity analysis
declared weight importance
HDI 2010 more coherent than HDI 2009
Non invasive Sensitivity analysis
declared weight importance
39
Two applications of non invasive sensitivity analysis