Jennifer Mulik - PACT, Kenya

March 19, 2013Jennifer Mulik, MA, MPH– Global Director [email protected]

Measuring Capacity Development Performance

The problem

“The monitoring and evaluation of capacity building is as much a challenge now as it was two decades ago.”

Simister & Smith, 2010, INTRAC

Theory of Change for Capacity Development

Outputs

Change in the internal systems, skills and policies of organizations, networks and systems

Outcomes

Improved external performance of organizations, networks and systems

Impact

Improved health, environment and/or livelihoods in the communities served by organizations, networks and systems

(a)Stronger local organizations;(b)Improve their performance; and (c)Have greater impact in their target communities.

• OCA• OD

Roadmap• ISP

Tracking

• OPI• Outcome

Mapping• Most Significant

Change

• Evaluations

Defining Performance - Framework

IDRC/Universalia CD Outcomes Framework

Efficiency

Relevance

Sustainability

Effectiveness

Leadership

Structure

Human resources

Program / services

Financial

Technology

Linkages

Administrative

Political

Social / cultural

Technological

Economic

Stakeholder

History

Mission Culture

Incentives Rewards

Infrastructure

Operationalizing Performance

Effectiveness:Results

Standards

Efficiency:DeliveryReach

Relevance:Target Population

Learning

Sustainability:Resources

Social Capital

Benchmarks

 Level 1 Level 2 Level 3 Level 4

Effective

Results

The organization is in the process of developing outcome level targets for its programs & services.

The organization has set clearly defined outcome level targets for its programs & services.

The organization has met over 50% of outcome level targets for its programs & services.

The organization has met over 75% of outcome level targets for its programs & services.

Level 1Level 2

Level 3Level 4

Global OPI Scores by Length of Partnership

Results

Standards

Delivery

Reach

Target Population

Learning

Resources

Social Capital

1.0

2.0

3.0

4.0

Less than 1 year

1-3 Years

More than 3 years

Global OPI Scores Time Series

Results

Standards

Delivery

Reach

Target Population

Learning

Resources

Social Capital

1.0

2.0

3.0

4.0

2011

2012

Ethiopia OPI Scores by Length of Partnership

Effectiv

enes

s

Efficienc

y

Relev

ance

Sustai

nabi

lity

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1.3

1.81.5

1.8

2.62.8 2.8

2.52.5

3.1 3.1

2.5

Less than 1 year1-3 YearsMore than 3 years

Organizational Performance Index (OPI) Testing Project

The purpose of the OPI Testing Project funded by the Rockefeller

Foundation is to check the OPI reliability, i.e. whether the Index generates the same set of scores when it’s reviewed by two people,

separately.

Kappa statistic inter-score reliability

Country Pact 1 v Pact 2

Pact 1 v Partner

Pact 2 v Partner

Vietnam .804 .498 .556

Ethiopia .706 .454 .576

South Sudan .605 .571 .437

Swaziland .504 .495 .583

Nigeria .415 .502 .299

Zimbabwe .357 .724 .587

Result

Our OPI tool and methodology of application

meets the reliability standard