Constructing an Index of Objective Indicators of Good Governance Steve Knack & Mark Kugler PREM Public Sector Group, World Bank October 2002 Introduction Different indicators of good governance are appropriate for different purposes. Indicators differ across (at least) two important dimensions. First, some indicators measure relatively specific aspects of the quality of governance while others are more highly aggregated. Second, some indicators are more transparently constructed and replicable, whereas others are less so – for example, subjective ratings provided by firms assessing political risks to foreign investors. Relevance for Bank operations requires the use of indicators that are as specific and disaggregated as possible. For other purposes, such as making broad comparisons across countries, or conducting research on the causes and consequences of good governance broadly defined, highly aggregated indicators are often preferred. For many purposes, researchers and donor organizations are free to use subjective assessments of the quality of governance constructed wholly without the cooperation or knowledge of developing country governments. In some cases, however, donors find that “ownership” of indicators by developing country governments is essential. Governments commonly object to use of broad, subjective assessments of corruption, political freedoms, etc. produced by TI, Freedom House and commercial firms assessing political risk. This note describes a methodology for constructing an index of objective indicators of good governance. The indicators were selected primarily with regard to broad cross- country coverage, and acceptability to developing country governments. Indicators available only for a small number of countries were avoided, as were indicators based wholly or in large part on expert opinion of westerners. This exercise is intended to provoke debate regarding the value of an index, and how one should be constructed, rather than to generate a final set of rankings. Although we believe there is merit in the particular set of indicators used here, we recognize that each indicator has its own idiosyncrasies and deficiencies, and we hope to gradually add to this set and replace some of the conceptually weaker indicators as more data become available. Rationale for an index The DAC criteria for indicators of good governance to potentially include in the MDGs specified that the number of indicators should be small. Because any single objective indicator tends to measure only a very small part of the institutional and governance environment, a large number of indicators is needed for a fair and accurate depiction. The only way to attain reasonable accuracy, while maintaining objectivity and keeping the number of indicators low, is to aggregate indicators into a smaller number of indexes.
22
Embed
Constructing an Index of Objective Indicators of Good ...
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Constructing an Index of Objective Indicators of Good Governance Steve Knack & Mark Kugler
PREM Public Sector Group, World Bank October 2002
Introduction Different indicators of good governance are appropriate for different purposes. Indicators differ across (at least) two important dimensions. First, some indicators measure relatively specific aspects of the quality of governance while others are more highly aggregated. Second, some indicators are more transparently constructed and replicable, whereas others are less so – for example, subjective ratings provided by firms assessing political risks to foreign investors. Relevance for Bank operations requires the use of indicators that are as specific and disaggregated as possible. For other purposes, such as making broad comparisons across countries, or conducting research on the causes and consequences of good governance broadly defined, highly aggregated indicators are often preferred. For many purposes, researchers and donor organizations are free to use subjective assessments of the quality of governance constructed wholly without the cooperation or knowledge of developing country governments. In some cases, however, donors find that “ownership” of indicators by developing country governments is essential. Governments commonly object to use of broad, subjective assessments of corruption, political freedoms, etc. produced by TI, Freedom House and commercial firms assessing political risk. This note describes a methodology for constructing an index of objective indicators of good governance. The indicators were selected primarily with regard to broad cross-country coverage, and acceptability to developing country governments. Indicators available only for a small number of countries were avoided, as were indicators based wholly or in large part on expert opinion of westerners. This exercise is intended to provoke debate regarding the value of an index, and how one should be constructed, rather than to generate a final set of rankings. Although we believe there is merit in the particular set of indicators used here, we recognize that each indicator has its own idiosyncrasies and deficiencies, and we hope to gradually add to this set and replace some of the conceptually weaker indicators as more data become available. Rationale for an index The DAC criteria for indicators of good governance to potentially include in the MDGs specified that the number of indicators should be small. Because any single objective indicator tends to measure only a very small part of the institutional and governance environment, a large number of indicators is needed for a fair and accurate depiction. The only way to attain reasonable accuracy, while maintaining objectivity and keeping the number of indicators low, is to aggregate indicators into a smaller number of indexes.
Aggregating tends to reduce measurement error. Indexes of several variables which all purport to measure a similar concept are in general more accurate than are their component variables. Each component variable reflects not only something about the quality of governance, but also idiosyncratic factors. For example, trade taxes as a share of all government revenues is sometimes sued as a proxy for administrative capacity, but it also may be affected by trade policy.1 As long as the idiosyncratic factors in each component variable are largely independent of each other, their effects on country rankings will be dampened greatly by aggregation. Index components The nine indicators we use are the regulation of entry, contract enforcement, contract intensive money, international trade tax revenue, budgetary volatility, revenue source volatility, telephone wait times, phone faults, and the percentage of revenues paid to public officials in bribes, as reported in surveys of business firms. A brief description of each component indicator follows:
Regulation of Entry: The number of procedures to start new businesses varies dramatically across countries. Some regulation is required on efficiency and equity grounds; however, the number of procedures required to start a new business, and the cost in time and fees, tends to be very low in many countries (such as Canada) in which social and environmental regulations are most stringent. The obstacles that an entrepreneur must surmount to open a new business in many countries far exceed anything that can be justified on efficiency grounds. Djankov et al. (2001) have collected data on the number of procedures that are officially required to obtain all necessary permits and completing all of the required notifications for the company to operate legally. For simplicity, the data collected apply to a “standardized firm” which operates in the largest city, performs general industrial or commercial activities, does not trade across national borders or in goods subject to excise taxes, is domestically owned, does not own land, etc. Contract Enforcement: Sometimes it is necessary administer the relationships between creditors and debtors to ensure equality, but the inability to enforce contracts without exceeding expense is indicative of overregulation. The indicator of contract enforcement refers to the number of formal independent procedures to collect a debt. The data pertaining to contract enforcement are derived from questionnaires answered by attorneys at private law firms. The current set of data refers to January 2002. The questionnaire covers the step-by-step evolution of a debt recovery case before local courts in the country’s largest city. The number of procedures covers all independent procedural actions, mandated by law or court regulation, that demand interaction between the parties or between them and the judge or court officer. Contract Intensive Money: Contract intensive money is the proportion of the money supply that is not held in the form of currency, i.e. the proportion that is held in bank
1 Higher import tariffs will increase trade tax revenues for a given level of imports, but may reduce revenues if they lower import volumes sufficiently.
2.
accounts and as other financial assets. The percentage of contract intensive money indicates in part how much faith investors have in the government's ability and willingness to enforce financial contracts, and to refrain from expropriating financial assets. It is a measure of trust in government and in banks, which are regulated by government. Contract intensive money is calculated as one minus the ratio of currency outside of banks to the sum of money and quasi-money (one minus line 14a divided by the sum of lines 34 and 35 in the IFS). International Trade Tax Revenue: Reliance on revenue from international trade taxes is widely believed to reflect weak administrative capacity. Economic theory suggests that taxing all transactions at low or moderate levels is more efficient than collecting taxes from only a subset of transactions at high rates. However, effectively collecting income, sales or other taxes on a broad range of transactions requires a certain degree of administrative capability on the part of governments. It is relatively easy for governments to collect tax revenues from cross-border transactions, because they are more easily monitored. Budgetary Volatility: Theory and evidence indicate that volatile and unpredictable government policy reduces private investment. The budget is one key arena in which government policy issues are played out, resulting in executive spending decisions. To the extent that policy decisions are captured in the budget, then stable policy should be reflected in stable budget allocations, and vice versa. Budgetary volatility is calculated using data from the most recent 4-year period on fluctuations in expenditure shares across the 14 functional classifications in the Government Finance Statistics data. Revenue Source Volatility: Volatile and unpredictable government revenue collection policy can discourage adequate long run planning. The manner and degree of revenue collection is an aspect of government policy determined in part by the executive. To the extent that policy decisions are captured in revenue collection policy, then stable policy should be reflected in stable revenue proportions, and vice versa. Revenue volatility is calculated using data from the most recent 4-year period on fluctuations in revenue shares across the 20 revenue classifications in the Government Finance Statistics data. Telephone Faults: The ability to provide and maintain consistent telephone service, or to regulate effectively private telecom industry, is an indicator of administrative capability. Access to telecommunication services helps to promote an environment conducive to business, and is necessary for businesses and households to take advantage of “E-Government” services. Telephone faults per 100 main lines is calculated by dividing the total number of reported faults for the year by the total number of main lines in operation and multiplying by 100. The definition of fault can vary. Some countries include faulty customer equipment. Others distinguish between reported and actual found faults. There is also sometimes a distinction between residential and business lines. Another consideration is the time period as some countries report this indicator on a monthly basis; in these cases data are converted to yearly estimates.
3.
Telephone Wait Times: See above for rationale. Waiting time is the approximate number of years applicants must wait for a telephone line. Percentage of Firm Revenues Paid as Bribes: Bribery and corruption are both a cause and a consequence of weakened governing institutions. Gauging the level of corruption that businesses face can provide information about the strength of governance in countries. The World Business Environment Survey (WBES) regularly asks businesses, “On average, what percentage of revenues do firms like yours pay in unofficial payments per annum to public officials?” The component indicator is the mean category of response within a country for 2000, the latest year available.
Methodology In constructing an index of objective indicators of good governance, the component indicators should be reasonably well correlated with each other. A standard statistical measure of index reliability, “alpha,” varies from a low of 0 to a maximum possible value of 1. Alpha is a positive function of (1) the mean inter-item correlation of the index components, and of (2) the number of index components. Our index is based on nine indicators, and the average inter-item correlation is about .25, producing a relatively high alpha reliability coefficient of .75. 2 All 36 of the inter-item correlations among the 9 component variables are in the expected direction, with the majority of these relationships being statistically significant at the .05 level. Controlling for per capita GDP tends to reduce the strength of these correlations: although most of them remain in the expected direction, only 7 of them are significant when the common effects of per capita income are statistically removed. Furthermore, it is encouraging that the indicators are correlated with other comprehensive measures of governance. The most encompassing measures of governance to date are the six “KKZ” (Kaufmann, Kraay, and Zoido-Lobaton, 2002) indexes, constructed from subjective assessments of governance. All 9 of the objective indicators are significantly correlated with each of the 6 KKZ indexes. Even after controlling for per capita GDP, 49 of these 54 correlations retain the expected sign, and 27 of them remain statistically significant. A strong relationship between two sets of indicators does not necessarily imply, of course, that either set is necessarily valid; however, the absence of such a relationship would have strongly suggested that one set or the other, or both, were not valid. These findings suggest that it is appropriate to aggregate these 9 objective indicators to construct a broader measure of governance. The first complication in aggregating the component indicators is that values for each of them are on disparate scales. To overcome this obstacle, each indicator is recoded to the standard normal distribution by
2 Factor analysis confirms that these indicators load primarily onto a single factor, indicating that they are measuring something in common. The only exception is the Djankov measure of contract enforcement (the number of independent procedures necessary to collect a bad check). Throughout the analyses reported below, however, there are no substantive changes in the results if this variable is omitted from the index.
4.
subtracting each value from its mean, and then dividing by its standard deviation. Standardization ensures that the rank and difference between countries is preserved and that each component in the index receives equal weight.3 The second complication in aggregating the component indicators is the potential for bias caused by the use of different data sources. Even if the values for all countries are accurate for a set of indicators, varying country coverage among the indicators could produce an inaccurate index. For example, countries ranked near the bottom on indicators constructed from GFS data (budgetary volatility and revenue source volatility) conceivably are not actually among the most poorly-governed countries in the world, but instead may just be the most poorly-governed among those with a reasonable capacity for statistical reporting. Countries without such minimum capacity could be rewarded, in effect, for their inability or unwillingness to report data. An index that includes some variables covering non-represented samples of countries could therefore contain bias. Our solution to this problem is to identify a subset of indicators that cover a representative sample of countries, and use values for those indicators to adjust the values for indicators with non-representative coverage.4 To determine which indicators cover representative samples of countries, we created a dichotomous variable for each component indicator that takes the value of 1 for any country for which there are data present for the indicator in any of the past five years, or a value of 0 if data are missing for all of the previous five years. Each of the 9 dichotomous variables was then regressed on the log of per capita income, using logit regression.5 Data availability was positively and significantly (using a .10 significance level) related to per capita income for 6 of the 9 indicators, and negatively and significantly related to contract-intensive money. Country coverage was representative (by income level) only for two indicators, telephone faults and telephone wait times. These two representative indicators were combined to form an unweighted index, ignoring missing values for either of the two components. This index, free of bias from non-representative country coverage, was then used to adjust the values for the component indicators with non-representative coverage, to keep from penalizing countries that are ranked poorly among a sample of countries biased toward those with stronger governance. This “percentile-matching” adjustment is done in the following way: (1) countries were ranked by their values on the non-representative indicator; (2) the same set of countries is ranked by their values on the index of two representative indicators, and; (3) each country’s adjusted score on the non-representative indicator is
3 Without standardizing, index components with higher means or variances are implicitly weighted more heavily in the index, even when no explicit weighting procedure is used. 4 We borrowed this percentile matching procedure from other work on governance indicators conducted within the World Bank by Aart Kraay. 5 The assumption is that a sample that is representative with respect to income will likely be representative in terms of the quality of governance. Data availability is regressed on the log of per capita income because it provided a better fit than per capita income. However, there are no substantive changes if per capita income is substituted for the log of per capita income.
5.
determined by matching its rank with the similar-ranking score on the index of representative indicators.6 This percentile-matching procedure requires that all countries with available data on the non-representative indicator also have available data on the index of representative indicators. Where data on the index of representative indicators were unavailable, values were estimated, from a regression of the index on the non-representative indicator.7 Index Validity Because most of the 9 objective indicators individually measure only very narrow aspects of the quality of governance, their partial correlations (controlling for per capita income) with the KKZ governance indicators tend to be modest. If the index accomplishes its purpose of reducing measurement error – reflecting idiosyncratic factors influencing each of the 9 component indicators – then its correlation with the KKZ governance indicators should be higher (assuming, again, that the KKZ indicators themselves are reasonably valid). Results provide support for this assumption. The average of the 9 correlations between the component indicators and each of the 6 KKZ indexes ranges from a low of .42 (for the KKZ Voice & Accountability index) to a high of .51 (for the KKZ Government Effectiveness index). Correlations of the index of objective indicators with the KKZ variables are much higher, as shown in the first column of figures in Table 1, ranging from a low of .55 for Voice & Accountability to .70 for Government Effectiveness. Controlling for per capita income, the average of the partial correlations of the 9 component indicators with each of the KKZ indexes ranged from only .13 for Voice & Accountability to .21 for Government effectiveness. The partial correlations of the objective index with the KKZ indexes are again higher, as shown in the figures in parentheses in the first column of figures in Table 1, ranging from .19 (for KKZ Political Stability and Control of Corruption) to .33 (for KKZ Regulation Quality).
6 For example, suppose a country is ranked 80th-best of 90 countries on a non-representative indicator. These 90 countries (and only these 90) are then ranked by their values on the representative indicator (ignoring the values and ranks of any other countries with data on the representative indicator for which data were unavailable on the non-representative indicator). The value for the 80th-ranked country on the representative indicator is then identified, and that value is assigned as the adjusted value of the non-representative indicator for the country ranking 80th on the non-representative indicator. 7 There are no substantive differences in the results when imputed values are left out of the analysis, but imputation allows many more countries to be included in the final index.
6.
Table 1: Correlation of Objective Governance Indicators with KKZ Governance Variables
(correlation values in parentheses control for per capita GDP)
Second Generation Indicators Percentile
Matching Index Unadjusted
Standardized Index Voice &
Accountability0.55*
(0.23*) 0.60*
(0.27*) Political Stability
0.63* (0.19*)
0.69* (0.24*)
Government Effectiveness
0.70* (0.32*)
0.77* (0.41*)
Regulatory Quality
0.61* (0.33*)
0.67* (0.38*)
Rule of Law
0.69* (0.31*)
0.76* (0.40*)
KK
Z In
dica
tors
Control of Corruption
0.62* (0.19*)
0.72* (0.36*)
Note: * Indicates correlation coefficient is statistically significant at the .05 level.
The right-hand column in Table 1 lists correlations between the KKZ indexes and an unadjusted version of the index of objective indicators, which standardizes and equally weights the 9 components but does not adjust for non-representative samples. These are higher in every case than the correlations with the adjusted index. The percentile matching procedure necessarily discards some information, which may weaken the associations with other variables somewhat. The problem is that the procedure does not preserve the relative distances between the scores of the non-representative component indicators, but preserves only the rankings and converts the relative distances to those represented in the representative components.
7.
Figure 1: Relationship Between Objective Indicators Index and KKZ Index
(controlling for per capita GDP)
Second Generation Index & KKZ Government Effectiveness(Controlling for Per Capita GDP)
KKZ Gov't Effective, resid-1.5 -1 -.5 0 .5 1 1.5
-1.5
-1
-.5
0
.5
1
BLR
PRY
TKM
RUS
VEN
DZA
AGO
KAZ
ROM
SDN
ALB
ECUMKDKWT
ZWE
AZE
GAB
ARMSYR
UKR
BRA
SAUMDACOL
HTI
TGO
GTM
ARG
COG
UZBKOR
TUR
DOM
TJK
BGR
ITA
PERIRN
GEO
SVK
PANARE
GNB
NIC
MLI
LKA
HRV
KGZ
SLVIDN
JPN
NPLZAF
PNG
GRC
POL
SVN
HNDMEX
CZEJAMMLTTWN
SLE
CIVTHAISRCYPLVAHUN
BOL
LTUHKG
MRT
LBN
NER
PRT
NGA
MYSPAK
GUY
PHL
NORBELFRABGDURYTTO
BHS
BDI
SUR
KEN
CHN
VNM
MUSCMRETHUSA
MAR
IND
LUXNZL
LAO
YEMCRI
ESTAUT
FJINAM
SWE
ZMB
EGYBLZ
DNKAUS
GHA
JORCAN
BWAMWI
FINDEU
UGAIRL
MOZ
ESP
CHL
GBRNLDISLCHE
MDG
SEN
GIN
TZA
BFA
MNG
GMB
TUNKHM
SGP
BEN
N: 142 t-statistic: 4.06 R-Squared: 0.541 Figure 1 depicts the relationship between the index of objective indicators and the KKZ indicator of governance effectiveness (again, controlling for per capita GDP). Appendix I provides similar graphs using the other 5 KKZ governance indicators. Collectively, the findings reported in this paper suggest that one can be reasonably confident that there is a good deal of validity in the index of objective governance indicators.
8.
References de Soto, Hernando. 1989. The Other Path: The Invisible Revolution in the Third World.
New York: Harper & Row. Djankov, Simeon, Rafael La Porta, Andrei Shleifer, and Florencio Lopez de Silanes,
“The Regulation of Entry,” World Bank Working Paper, June 2001. Djankov, Simeon, Rafael La Porta, Florencio Lopez de Silanes, and Andrei Shleifer,
“Legal Structure and Judicial Efficiency: The Lex Mundi Project,” World Bank Working Paper, October 2001.
Kaufmann, Daniel, Aart Kraay, and Pablo Zoido-Lobaton. 2002. “Governance Matters
II: Updated Indicators for 2000/01.” World Bank Policy Research Working Paper 2772.
9.
Appendix I: Graphs of the relationship between the objective indicators index and KKZ governance indexes (all graphs control for per capita GDP)
Second Generation Index & KKZ Voice and Accountability(Controlling for Per Capita GDP)