Vladimir Eliseenko head of NPM technologies department

Methodology and practical experience in elaboration of aggregated ratings and

indexes, measuring public management quality and efficiency

Vladimir Eliseenko

head of NPM technologies departmentHSE Institute for Public administration and public management

executive secretaryHSE coordination group for strategic development in PA research and teaching

Higher School of Economics , Moscow, 2011

www.hse.ru


Focus on indexes/ratings/grading/ranking in Russia (1)

photo

photo

photo

“Conception (program) of administrative reform in Russia Federation (2006-2010)” contained self-identification and target-setting toward 3 international indexes:

Governance Research Indicator Country Snapshot - GRICS (World Bank)

Global Competitiveness Report (World Economic Forum)

Corruption Perceptions Index (Transparency International)


Focus on indexes and ratings in Russia (2)

photo

photo

photo

Rankings and ratings, measuring new technologies of public administration and public management

I.Quality and scope of NPM technologies applied by subfederal governments (Ministry of Economic development/ Higher school of Economics)

II.Quality of financial management (Ministry of finance)

III.Quality of government cervices (Ministry of Economic development / Higher School of Economics)

IV.Quality of government bodies’ web-pages and e-services Administrative reform (Higher school of economics among 10 others)


Focus on indexes and ratings in Russia (3)

photo

photo

photo

Rankings and ratings of subfederal and municipal authorities according with President’s decrees №607 and №825

More than 230 performance measures for subfederal bodies (governors) and more than 140 performance measures for the heads of municipal authorities

Collected each year (actual for two latest years and predicted for three next years)

Covers both outcome measures and measures for government activities and services

Different measures provided by different sources (official statistics, data from federal and subfederal governments, own information, citizen surveys)

There’s obligation to construct ratings (Ministry of Regional Development for governors and, accordingly, subfederal governments for the heads of municipal authorities) and to give financial grants to best performers


Life after index – how to turn measurement into management (1)

photo

photo

photo

Current problems:

1) frustration with the results (dynamics) of Russia in international indexes:

- are we doing not enough?- are we going the wrong way?- maybe we need more time to see the consequences and to gain the results?

2)what’s the best decision(s) based on the index results?- is it correct to medal the governor, mayor or federal authority who is best “in average”- how we can indicate the “best practices” i.e. define what’s really useful in our effort to improve the quality of life?-if we focus on outcome measures – we can not exclude inertia (- contradiction “max competition – max similarity”



photo

photo

photo

Factor 1. Ensure the need Designing performance management systems to

fulfill multiple goals

Establishment of “agent of change”, capable of and

responsible for managing the elaboration of performance

measures and to coordinate the collection, analysis and use of

performance information

Factor 2. Provide a plan

Factor 3. Build internal support for change and overcome resistance

Factor 4. Ensure top-management support and

commitment

Factor 5. Build external support

Factor 6. Provide resources

Factor 7. Institutionalize change

Factor 8. Pursue comprehensive change

Overcoming measurement

challenges

Designing effective incentives

Managing the performance of

independent agents

Applying comprehensive implementation

strategies

Imposing the system of performance measures,

ensuring their quality and utility for a given purpose

The connection and use of additional benchmarking data

consumers

Dealing with organizational issues to create sustainable and effective benchmarking system

Defining the relations between performance benchmarking and performance contracting

Fernandez/Rainey (2006)

Forsythe (2001)Eliseenko (2007)



photo

photo

photo

Decision 1. Compare units on the basis of dynamics, not on the basis of absolute values

There’s too much economic and social differentiation in Russia, both across the regions and inside the regions (across)

It allows to compare the whole range of units without dividing them into subgroups (amplifies the competition)

It results in the “victory of the weak” but it is wonderful from the point of:

a) equalization (levelling)b) bringing the attention to the system



photo

photo

photo

Decision 2.Connect the weights of indicators (measures) with the difference of values across units

eliminates the effect of levelling (average of average)

shows clear priorities

stimulates to concentrate the efforts on “hard” measures not on the easiest to achieve



photo

photo

photo

Decision 3. Find the cause-effect chains across different levels of performance measures



photo

photo

photo

Decision 3. example of cause-effect chains across different levels of performance measures

Education Performance-based budgeting

“Money follows the pupil”

Financial grants for best teachers

Qualification of teachers

Percentage of interactive lessons

Unified testing results

Percentage of pupils who hasn’t got a certificate

Public housing and utilities

Performance-based budgeting

Subsidies to low-income families instead of losses compensation

Financial health of service providers (both private and public)

Quantity of complaints on the poor quality of the absence of services (water, heating etc)


Life after index – how to turn measurement to management (7)

photo

photo

photo

Decision 3. example:

SCHOOLEDUCATION

SCHOOLEDUCATION

HOUSING SERVICES AND UTILITIES

HOUSING SERVICES AND UTILITIES

PUBLIC HEALTHPUBLIC HEALTH

AOAO AOAO AOAOAEQAEQ AEQAEQ AEQAEQIII III III


Life after index – how to turn measurement to management (8)

photo

photo

photo

Decision 3. possibility to prove (by econometrics) the cause-effect chains across different levels of performance measures


Ongoing project: data (1)

photo

photo

photo

We use HSE ranking, based on the quality of administration in regional executive bodies (mostly the components of administrative reform) – measured in 2006 and 2007

А+ Excellent quality of PA

А Good quality of PA

В Middle quality of PA

С Poor quality of PA

Performance management and performance based budgeting (programming, internal and external, planning, evaluating the efficiency of government expenditures and government agencies and NDPB)

Quality of internal management (functional analysis, optimization of government regulation and control, anticorruption activities,

Treating citizens as clients/consumers (administrative simplification, quality of government services, openness and participation, “single window” and one-stop shops for government services, e-services)


Ongoing project: data (2)

photo

photo

photo


Ongoing project: hypotheses (1)

photo

photo

photo

Main:

G1. The implementation of new PA tools and technologies influence the quality of regulation and administration in different branches of government policy. G1A. The degree of influence for each new PA technology can be different, depending on actuality (starting position).

G2. The quality of regulation and administration in a certain branch of government policy positively influence the efficiency and quality of private firms and government establishments, providing the services to the citizen and businesses

G3. The quality of services to the citizen and businesses positively influence the dinamics of outcomes in a certain branch of socio-economic system.


Ongoing project: hypotheses (2)

photo

photo

Alternative:

AG1. Quality and speed of adoption of new PA tools and technologies is not independent but dependent variable. It depends on the qualifications of higher-level management in subfederal government. Thereby, the participation in federal PA reforms distracts human and financial resources, which could be spent more effective way.

AG2. Only economic variables (per capita GDP, per capita budget revenues) can significantly influence the outcomes (the quality of living) and government efficiency.

AG3. Government is an evil itself so the reduction of public servants in any situation can positively influence the outcomes (quality of living)

Vladimir Eliseenkoe-mail: [email protected]

room k-20420, Myasnitskaya str., Moscow, Russia,

Phone.: +7 -916-596-6620

Vladimir Eliseenko head of NPM technologies department

Documents

ratings of subfederal

performance measures

public management quality

outcome measures

international indexes

servicesdifferent measures

index results

public managementquality