Developing a National Framework for Monitoring the Iraqi NDP MEASURING GOVERNANCE Marie Laberge Global Programme on Country-Led Governance Assessments.

Developing a National Framework for Monitoring the Iraqi NDP

MEASURING GOVERNANCE

Marie Laberge Global Programme on Country-Led Governance Assessments

UNDP Oslo Governance Centre

Examine the limitations of global composite indicators

Discuss the rationale for using complementary indicators (input vs. Process vs. outcome – de jure vs. de facto)

Identify indicators that are ‘actionable’

Identify 4 ways in which indicators can be made pro-poor and gender sensitive

Identify common sources of governance data

Types of indicators & data sources

Part OneInternational sources of governance indicators

90888682 84 96 98 00 02 0492 94 0678761974

Global composite indicators of governance

0880

CPIA Freedom in the World

Commitment to

Development Bertelsmann Transformation

Index

Global Accountability

Report

Index of Economic Freedom

Journalists killed

Open Budget IndexPolity

International Country Risk

Guide

Corruption Perceptions

Index

Opacity Index

Integrity Index

Human Rights

Indicators

Bribe Payers Index

GAPS in Workers’ Rights

BEEPS

Press Freedom Survey

Political Terror Scale

Global Competitivenes

s Index

World Governance Assessment

World Values Survey

State Failure Dataset

Weberian Comparative State Project

CIRI Human Rights

Databse

Women in Parliament

Governance Matters

Gender Empowermen

t Measure

Index of Democracy

World Democrac

y Audit

Countries at the Crossroads

Failed States Index

Press Freedom

Index

Democracy Index

Institutional Profiles

Database

Governance and

Democracy Processes

Global Peace Index

Transparency International

Freedom House World Bank Governance Indicators

Bertelsmann Foundation

Global Integrity

(Syria, Saudi Arabia not included)

Top JordanSaudi ArabiaKuwaitMorocco

“Partially free”: Jordan, Morocco, Lebanon, Yemen, Kuwait

Kuwait, Jordan, Saudi Arabia, Morocco

Lebanon, Kuwait, Algeria

Jordan, Kuwait, Egypt, Palestine

Bot-tom

LebanonYemen

“Not free”: Algeria, Egypt, Saudi Arabia, Syria Palestine

Syria Yemen Palestine

Syria Saudi Arabia

MoroccoAlgeria Yemen Lebanon

1) What is each index measuring? TI CPI Public sector only WB ‘Control of Corruption’ indicator public &

private sectors Global Integrity Index anti-corruption mechanisms

2) What types of indicators make up each index?

High-level corruption vs. petty corruption Frequency of bribes vs. size of bribes

How to explain these discrepancies in rankings?

Selected sources

(out of 13 in 2009)

Subject asked (based on

perceptions)

Respondents Coverage

World Economic Forum (WEF)

Undocumented extra payments or bribes

connected with various government functions

Senior business leaders; domestic and international

companies

131 countries

Bertelsmann Transformation

Index (BTI)

The government’s capacity to punish

and contain corruption

Network of local experts / local

business people

125 countries

Economist Intelligence Unit

(EIU)

The misuse of public office for private (or political party) gain

International experts

170 countries

3) What data sources are used?

And… 4) Are data sources the same

every year?

TI Corruption Perception index (CPI) 2009 126th Syria 5 surveys

176th Iraq 3 surveys

179th Afghanistan 4 surveys

2008 147th Syria 5 surveys

176th Afghanistan 4 surveys

178th Iraq 4 surveys

2007 138th Syria 4 surveys

172th Afghanistan 4 surveys

178th Iraq 4 surveys

Part twoNational governance indicators

Monitoring the input, process and outcome sides of governance reforms

Outcome indicators will tell us:

What do we want to achieve?Have we achieved it?

Process indicators will tell us:

How are inputs used to generate outcomes?What are the ‘best’ processes?When & how do processes need adjustment?

Input indicators will tell us:

What is needed to create these processes?Whether it is available?

Three levels of indicators

• De jure (input) indicator: In law, is there an agency with a legal mandate to address corruption? Does it receive regular funding, a professional & full-time staff?

Measuring a change in law, in procedures, in resources

• De facto (process / outcome) indicator: In practice, is the anti-corruption agency effective?

Measuring improved governance in practice (how are citizens benefiting from this new institution?)

Complementarity in the use of indicators

Combining indicators to show discrepancies De jure indicator: In law, is there an agency with a legal mandate to address corruption? (input)

De facto indicators: In practice, is the anti-corruption agency effective?

(outcome) • When necessary, is the ACA able to independently

initiate investigations? (process) • Does the ACA make regular public reports (e.g. to the

parliament)? (process) • Can citizens complain to the ACA without fear of

recrimination? (e.g. whistle-blower mechanisms) (process)

• Does the ACA act on citizen complaints within a reasonable time period? (process)

• Outcome indicators are useful to assess progress towards the desired objectives of governance programmes (new laws / increased expenditures mean nothing in and of themselves)

• But they lack ‘actionability’ (they don’t tell us what needs to be fixed / points of interventions)

• Example: A government cannot ‘choose’ to lower a crime rate (an outcome indicator)

• It can, however, choose to put more police on the streets (input) or toughen penalties for offenders (prosecution process)

Input / process indicators are more ‘actionable’

Outcome indicators (non-actionable)

Input / process indicators (actionable)

% citizens who believe there is a lack of transparency in the operations of local governments

Number of provinces which have a public forum for citizens to discuss with locally elected officialsNumber of provinces which formally publish contracts, tenders, local budget and local development plan

% citizens who do not believe that legal protection is ensured equally to all citizens regardless of their material status, ethnic, religious affiliations, political/party affiliations

Number of provinces where an awareness-raising programme on citizens’ rights to seek legal protection has been conducted% citizens who say they know where to / how to seek legal protection

Actionable indicators:

Experience from Morocco

“Not everything that counts

can be counted, and

not everything that can be

counted counts.”

– Albert Einstein

A word of caution:

“Not everything that counts can be

counted, and not everything that can be

counted counts.”– Albert Einstein

Measuring is not an end in itself, but rather a means to an end (actual governance reforms).

So what we measure should help us implement reforms – if not, why

measure it?

A word of caution:

• Risk of measuring things because they are easily measurable, leading to ‘reform illusion’

• Example: Measuring the number of corruption cases brought to trial (as an indicator of the efficiency of the judicial system in combating corruption):

• Does an increase in this indicator mean an increased level of confidence in the reporting mechanism, and in the courts?

• Or rather, does it indicate a higher incidence of corruption?

• Or both...?

In other words, are you really measuring what you intended to measure?

A word of caution

Balanced baskets of indicators

Do you have a balanced basket of indicators that measures progress towards a single aim?

Is the ambiguity inherent in each indicator reduced by the presence of the others?

Balanced baskets of indicators

– What is the aim you want to achieve?

– What are the many reasons why your aim might NOT be achieved?

– For each “reason” develop one indicator

Unbalanced baskets of indicators

Aim: “Equal access to justice” Indicator 1: Number of new courts opened in rural and urban areas Indicator 2: Number of courts per 100,000 residents Indicator 3: % citizens who say that they have access to court systems to resolve disputes  Indicator 4: % accused not represented at trial

Indicator 1: Number of new courts opened in rural and urban areas Measures the activity of a court-building programme, but does not tell us the result of the programme: do people have their fair share of courts?

Indicator 3: % citizens who say that they have access to court systems to resolve disputes Tells us nothing about equality

Indicator 2: Number of courts per 100,000 residents

Tells us nothing about the equality of the distribution of courts (serving mainly one group / one region?)

Indicator 4: % accused not represented at trial

The basket as a whole is unbalanced: 3 indicators relate to the courts, and one to legal aid

Balanced baskets of indicators

Checklist for building a well-balanced basket of indicators

  Avoid narrow activity indicators What is the strategic aim you want to

achieve? Draw one indicator from each

institution/service that contributes to the overall aim

Or...

What are the many reasons why your aim might NOT be achieved?

For each “reason” develop one indicator

Balanced baskets of indicators

4 key services/institutions for achieving “equal access to justice”

4 indicators:1. Court system

2. Legal aid service3. Police service

4. Prosecution service

Indicator 1 – Court system% citizens who say that they have access to court systems, disaggregated by gender, ethnicity, region, etc. Adapted from version 1: Disaggregated to reveal issues of inequality

Indicator 3 – Police service% citizens who say that the police will respond to them without requiring a bribe if called to resolve a dispute, disaggregated by gender, ethnicity, region, etc. Adds balance: Will alert you to problems in police services that may block access to justice more for some groups than for other.

Indicator 2 – Legal aid service% accused persons legally represented at one or more court appearances, disaggregated by gender, ethnicity, region, etc. Adapted from version 1: Disaggregated to reveal issues of inequality

Indicator 4 – Prosecution serviceRatio of prosecution caseloads in courts serving wealthier communities to those in courts serving marginalized communities Adds balance: Will alert you to problems in prosecution services that may block access to justice more for some groups than for other.

How to make governance indicators ‘pro-poor’ and gender sensitive

What makes a governance indicator ‘pro-poor’ or

gender sensitive?

4 ways to make indicators sensitive to vulnerable groups:

1. Disaggregating by poverty/gender2. Specific to the poor/women3. Implicitly poverty/gender

sensitive4. Chosen by the poor/women

1. Disaggregating by poverty/gender (Parliament)

Information is collected for the general population, then disaggregated by sex or income

• % seats held by women in parliament

• Level of confidence among female citizens that the Parliament represents their interests

• % of Parliamentarians from poorer districtsthat have functioning and accessible localoffices to meet with constituents

2. Specific to the poor/women

Measuring governance issues that are specific to the poor or to women

• Size of funds allocated to legal aid in provincial budgets (per capita)

• Number of attorneys as % of citizens in need of one

• % of local governments practicing gender-sensitive budgeting

3. Implicitly poverty/gender sensitive

• Backlog of small cases of little financial value Makes no explicit reference to poverty status or gender, but by its nature, clear that the indicator is of particular relevance to low-income groups / women

• The number of hours that polling booths are open during on election day

• Frequency of engagement of CSOs in consultations on the legislation-making process

4. Chosen by the poor/women

Identified & measured by using participatory techniques (surveys, focus groups, etc.)

• Acceptance of documentation other than birth certificates in the process of voter registration

• Women’s trust in the police and its ability to provide women with redress if they file a complaint

Sources of governance data

• Important to monitor both the ‘supply’ and ‘demand’ for good governance

• Parallel with private sector: – Assessment of the company’s charter, balance

sheet, internal processes and assembly line OR / AND ? – Assessment of whether consumers are actually

buying their product, whether they were satisfied with it, and likely to keep on buying it

Complementarity of data sources

Implications for data sources?

1) Objective data – Qualitative:

Administrative sources: Policy and legal documents, codes of conduct, organizational set-up and management systems, processes for decision-making

Other narrative reports: Newspaper reports, reports by CSOs

2) Objective data – Quantitative: Government statistics, data gathered by domestic NGOs, international organizations and academics, expenditure tracking and budgetary information, audit reports, court records

3) Opinion and experience-based data from citizens, specific groups (MPs), specific institutions, or private sector through surveys, focus groups, etc.

Assessing the ‘supply side’ of governance

Assessing the ‘demand side’ of governance

IHEC has a permanent voter registry with safeguards to prevent fraud and mechanisms for inclusion of all Iraqis in elections

-% of people prevented from registering (experience-based)-% of people prevented from voting because name not on voter list (experience-based)-% of people prevented from voting due to intimidation (experience-based)-% of people who say results accurate reflection of vote-% of people who feel elections produce legislature representative of people-% of people who feel elections enable them to get rid of bad leaders

Assessing the ‘supply side’ of governance

Assessing the ‘demand side’ of governance

Number of criminal codes and criminal procedure codes harmonized with international Human Rights Law

% who say ordinary people can commit crimes without punishment

% who say high level officials can commit crimes without punishment

% who say people fear wrongful arrest

Opportunity to draw from ‘fresh’ survey data

New ‘Arab Democracy Barometer’

Joint initiative by Arab Reform Initiative & Arab Barometer

Nationally representative public surveyIn 10 countries – including IraqData to be collected by country teams (starting autumn 2010)

75 questions

The Arab Democracy Index• First governance index produced by an Arab institution (“The

Arab Reform Initiative”) • Data collected by consortium of Arab research centres &

universities • Covers 10 countries• 40 indicators

Some measure the ‘means’ of democratic transition (legislation) – ‘de jure’ indicators

Some measure the ‘results’ of democratic transition – ‘de facto’ indicators

• Sources of data: Government & non-government Citizen’s impressions (public survey) used for 25%

indicators• 4 themes:

1) Strong & accountable public institutions2) Rule of law3) Respect for rights & freedoms4) Equality & social justice