USAF Capacity Building- Monitoring and Evaluation

MONITORING & EVALUATION

for

UNIVERSAL SERVICE

PROGRAMS/PROJECTS

Introduction

•  Personal –  Former CEO USF Pakistan –  First employee - built USF Pakistan

and ran it successfully for nearly 5 years –  Now a consultant in several countries of Asia and Africa

•  This Webinar –  NOT about M&E itself –  Rather about M&E in USOF Projects & Programs! –  After an introduction to the concept of M&E, “setting the stage”:

how to set up M&E in a Universal Service Organisation

Agenda

•  Part-1 –  Session 1- Setting the stage –  Session 2- Monitoring –  Session 3- Evaluation

•  Part-2 –  Session 1- Organizational Capacity Assessment –  Session 2- Sourcing Options –  Session 3- In-House M&E –  Session 4- Outsourced M&E –  Session 5- Procurement – if outsourced –  Session 6- Worked Example for Bid Evaluation

SETTING THE STAGE Part 1, Session 1

Project life

Focus of Monitoring

Focus of Evaluation

Goals Objectives Outputs Inputs Outcomes Impacts

Definitions

INDICATOR - a metric for measurement of inputs,

processes, outputs, outcomes, & impacts for

projects, programs, or strategies.

INPUTS - the main resources required to

undertake activities & to produce outputs.

OUTPUTS - physical &/or tangible goods / services delivered by the project.

Describes scope of the project

OUTCOME - what the project intends to accomplish by the end of it’s implementation.

IMPACT - result of the project, accrued in medium to long

term.

MONITORING Part1, Session 2

PLAN

Planning Phase Implementation Phase

PROGRESS

Monitoring is an ongoing process by which

stakeholders obtain regular feedback on the progress

being made towards achievement of their planned

objectives.

Are we taking the actions we said we would take?

Are we making progress on achieving the results that we said we wanted to achieve?

Monitoring should be able to respond to 2 questions

…all this is measured thru Indicators.

Examples of Indicators

– Funds spent in each region divided politically, geographically, etc

– No of contracts awarded

– Subsidy amount disbursed

– % of operational expenditures against contribution collected

Org

leve

l Pr

ojec

t lev

el – Telephone lines per 100 persons

– Computer usage rate in intervened area

– Mobile subscribers per 100 persons (segregated geographically)

–  Internet availability per 100 persons (based on technology)

– Volume of local content developed

What to Monitor?

Quality

Risk

Scope

Contracts

Schedules

Human Resources

Other factors

Who Monitors?

Monitoring Tools

Project

•  Periodic Reports

•  Computer based project management tools.

•  Network performance reports

e.g: network usage report, billing info,

QoS report, call success ratios, etc.

•  GIS Maps

•  Tech. Parameters validation through drive

test & other tech. metrics

Basic Design of Monitoring Framework

What is to be measured –indicators?

How it will be measured & recorded?

Who will measure the information?

How frequently info

will be measured &

reported?

EVALUATION Part 1, Session 3

What is Evaluation?

EVALUATION - a thorough & independent

assessment of activities to determine the extent to

which they are achieving stated objectives.

It is a periodic, objective assessment of an

ongoing or completed project, program, or

policy

Parameters of Evaluation

It relates to specific

questions regarding

design, implementation

and results.

It is carried out at discrete

points in time, often

seeking perspective of

outside experts.

Types of Evaluation – broadly

Program Performance Evaluation

addresses issues like

•  relevance to overall goal of org.

•  efficiency in delivery of inputs,

•  effectiveness of approach,

•  timeliness of approach

•  other questions regarding project/program design, management & operational decision making

Outcome/Impact Evaluation

•  investigates nature of relationship between planned inputs & outputs, & their outcomes

•  assesses, what difference (impact) the intervention made

•  and how much of the impact is attributable to the Program/Project.

When Evaluations are conducted?

Program / Project

for���long-term���Impact���Analysis

Post Evaluation

Interim Evaluations

Baseline Evaluation

Start Later ? End

When is the Post Evaluation conducted?

Whenever sponsors want to address question like:

–  In case of a Pilot project: What would be potential development contributions of an innovative new program (before it is started)?

–  In case of a successful small-scale Program: To what extent & under what circumstances could it be replicated on a larger scale or with different population groups?

–  In case of a completed Program: What has been the contribution of the intervention (made thru the Program)?

How to conduct Impact Evaluation?

Start  Time   T 0 Close  Time   T C

 Project  Ben

eficiary  

Evalua8on   E 0 Baseline

-­‐ Evalua8on  E C Post  Evalua3on

Evalua8on  could  cover • Demographic • Socio -­‐ Economic • Technology • Usage  pa@ern • Other  relevant  indicators

Impact=  Ec  –  Eo

Approach  -­‐  1  

Also called “Before & After method”

How to conduct Impact Evaluation?

Start    Time T 0 Close  Time T C

 Project    

 Ben

eficiary  

Project  

Non

-­‐Ben

eficiary    

Evalua8o

n  Ew

ith  

-­‐

Impact=  Ew

ith  -­‐  Ew

ithou

t

Evalua8on  could  cover • Demographic • Socio -­‐ Economic • Technology • Usage  pa@ern • Other  relevant  indicators

Approa

ch  -­‐  2  

Evalua8o

n  Ew

ithou

t    Evalua8o

n  Ew

ithou

t  

Also called “With & Without method”

Evaluation Tools

Project

•  Surveys

•  Beneficiary Assessment

•  Econometric Analysis- Regression

•  Case Studies

•  Cost Benefit Analysis

•  Cost Effectiveness Analysis

Info. required for Impact Evaluation (example Telecenter)

General information about area

Mapping of public & private facilities (schools, hospitals, post offices, …)

Demographic Info. (No. of persons - age group & gender segregated)

Number of persons who can access computers (gender-wise)

Cost per use of computer, PCO, & other services Persons using computer for educational purpose, health, others

Persons using computer for web based earning,

Average distance & time from nearest computer center

Other uses of computers & ICTs

Number of computers per 100 population

Number of persons who can access computer and internet

Number of persons who know how to use computer

Info. required for Impact Evaluation (example Telecenters)

Information on Purpose of using computers Info on Internet & Computers for learning Info on status of E-commerce Info on status of E-Health Info on status of E-governance Info on status of E-Agriculture Info on use of computer in social networking

Info on economic activities associated with internet & computer

Info on status of networking

Info on presence of local content in cyber space

Info on financial sustainability

Info on community participation in management of center

PART-2

ORGANIZATIONAL CAPACITY ASSESSMENT TEST – One of the ways!

Part 2, Session 1

Comparison of 3 scenarios for M&E implementation

Scenario 1

Sufficient capacity exists within the org.

•  Independent M&E unit

•  Capacity to execute M&E

•  M&E plan is available

•  Necessary staff available

•  Necessary hardware & software are available

•  Office-space is available

Scenario 2

Capacity exists but not sufficient

•  Capacity to execute M&E but with gaps

•  M&E plan not developed

•  M&E team deficient in terms of HR

•  Deficiencies in hardware & software

Scenario 3

Substantial Capacity gaps

•  No M&E set up of any kind

•  No dedicated human resource

•  No infrastructure

OCAT*- Sample Test

Question-1 Score What is the organizational structure in context of M&E?

Response Scores A separate M&E Unit with clear reporting lines existing, indicate a strong M&E setup viable for in-house M&E

2

A dedicated M&E human resource with no clear reporting line implies a weak M&E system within an organization

1

No M&E system within an organization structure 0

* Organizational Capacity Assessment Test

(… cont.) OCAT- Sample Test

Question-2 Scores What is the current M&E setup - human, financial, and technical resources?

2.1 Human resources 2.2 Budget (financial allocation) for M&E 2.3 Equipment available for M&E

Response Scores Sufficient 2 Needs Improvement 1 Non-existent 0

(….. Cont.) OCAT- Sample Test

Question-3 Score What is the rationale for creating M&E within the organization (Management’s perspective)?

Response Scores Clearly defined objectives, and policy exists regarding M&E 2 Management aware about M&E’s importance but no written policy or objectives exist

1

No information available 0

(….. Cont.) OCAT- Sample Test

Question-4 Scores What skills exist within the organization in the following areas;

4.1 Domain Knowledge of Monitoring & Evaluation 4.2 Experience of working and demonstrated capability of managing M&E

4.3 Designing and conducting research/surveys 4.4 Data collection, management, and reporting 4.5 Statistical Analysis 4.6 Knowledge about econometrics 4.7 Using research findings for decision making through recommendations

4.8 Project management 4.9 Development of computer based M&E system

Response Scores Sufficient 2 Needs Improvement 1 Non-existent 0

(….. cont) OCAT- Sample Test

•  Scoring Recommendations

–  Maximum attainable score = 28

–  Score above 90% indicates sufficient capacity for In-house M&E

–  Score between 60-89% indicates capacity, which need to be built

–  Score less than 60% indicates weak capacity, where Outsourced M&E is the most plausible option

SOURCING OPTIONS Part 2, Session 2

M&E Sourcing Options

OCAT Results

Sufficient Cap. In-house

Insufficient Cap.

In-house with Cap. Building

Cap. Gaps Out-source

Steps for M&E implementation in the 3 scenarios

Scenario 1 Scenario 2 Scenario 3

Implement M&E in-

house if not already

being done

Implement M&E in-house, but: •  Acquire deficient HR •  Develop M&E team •  Develop capacity of team thru

professional training •  Procure necessary HW & SW •  Develop M&E Plan •  Execute M&E Plan •  Outsource functions which are

not feasible or are occasional

Outsource M&E by

engaging a professional

firm

How to decide?

Build In-House Outsource

Possible reasons of selecting one over the other

•  Cost - less expensive to outsource (eg: there isn’t lot of work)

•  Lack of in-house capacity too big

•  Outside Providers’ research & specialized know-how exceeds by far

•  External validation is required

•  To keep in-house manpower small

•  A legal requirement?

•  Notion that third-party M&E is considered more “neutral”

•  Cost - may be less expensive in long run

•  Required expertise is available

•  Want direct control over the process

•  Desire of better quality

•  Flexibility in managing system configuration

•  Reliable & competent service providers not available

•  Routine nature of task

In-house Out-sourced

IN-HOUSE M&E Part 2, Session 3

Structure of a good M&E Unit - Functions

Manager

Survey

Enumerators

Monitoring

Engineers Field Coordinators

Database Coordinators

Evaluation

Data Analyst

Reporting Sociologist/ Economist

HR Requirements for In-house M&E (pg 1 of 2)

S. No Designation Experience Qualification No.

1 Manager

10+ years in Project Management + Monitoring & Evaluation of similar projects in development sector

Masters Level Degree — Management, Social Sciences, & Engineering

1

2 M o n i t o r i n g Specialist 5+ years in Monitoring

Masters Level Degree — Management, and Social Sciences 1

3 E v a l u a t i o n Specialist

5+ years of conducting Evaluation

Masters Level Degree — Social Sciences 1

4 S u r v e y Specialist

5 + years in conducting surveys

Masters Level Degree — Management, and Social Sciences 1

5 R e p o r t i n g Specialist

(5+ years in Report-writing

Masters Level Degree — Social Sciences (economics/sociology or other relevant disciplines)

1

Pg 2 of 2) HR Requirements for In-house M&E S. No Designation Experience Minimum qualification Nos.

6 Soc io log is t / Economist

5+ years of in capturing socio-economic aspects of development interventions

Masters Level Degree — Social Sciences (economics/sociology or other relevant disciplines)

1

7 Engineers 3+ years Bachelors Level — Engineering Telecom

Workload dependent

8 Data Analyst 3+ years in conducting data analysis & interpretation of data

Masters Level Degree -- Management or Statistics 1

9 D a t a b a s e Coordinator

5+ years in developing & managing databases

Masters Level Degree — Computer Sciences, Database Management

1

10 F i e l d coordinators

3+ years in field coordination

Bachelors Level Degree -- Sociology + other relevant disciplines

Workload dependent

11 Enumerators 3+ years in enumerating surveys

Bachelors Level Degree -- Sociology + other relevant disciplines

Workload dependent

Material Requirements for In-house M&E

S.No Item Quantities

1 Hardware

2 Software

3 Drive Test Equipment

4 Communication Equipment

5 Furniture

6 Vehicles

7 Office

8 Misc.

OUTSOURCED M&E Part 2, Session 4

Selection Process to outsource M&E

Invitation to prequalify

Receipt of applications from Firms

Evaluation of applications

Shortlist of the pre-qualified

RfP to the short-listed

Evaluation of proposals

Selection of the Out-sourcing Firm!

Task List for the Consultant/Firm

•  Develop a detailed configuration of M&E unit incl: identification of HR, job descriptions, competency levels & organization structure

•  Develop M&E Plan to address: What will be done? How it will be done? Who will do it? When it will be done? How much resources are required?

•  Build capacity of the staff for M&E;

•  Develop computer based M&E reporting system, along with user manuals & documentation;

•  Develop M&E Operational Manual for the staff;

•  Install hardware & software for M&E & train operators;

•  Train __ persons to conduct/supervise data-collection process;

•  Design & conduct baseline surveys in project areas;

•  Develop impact assessment methodology and & conduct impact assessment;

(.. Cont) Task List for the Consultant/Firm

•  Identify problems with M&E system & come up with solutions to those problems;

•  Develop transition plan to support USF in taking over M&E

•  Provide refresher training on M&E to USF staff;

•  Oversee design & conduct of various M&E activities by USF;

•  Review the results of studies undertaken & propose improvements;

•  Assist in data analysis & interpretation;

•  Review the system & propose amendments if required;

•  Provide technical support for maintaining computer based M&E system.

•  Ensure that staff is receiving adequate support to implement M&E functions & that data collection & analysis is on schedule

If out-sourced, some HR would still be required

At least with 5 basic skills:

1)  Project Management

2)  Contract Management

3)  Financial Controlling

4)  Technical Monitoring

5)  Socio-Economic Monitoring

Min HR required to monitor Out-sourced M&E

S. No Designation Experience Min. qualification Nos.

1. M a n a g e r M&E

5+ years in project management

Masters level —Management, Social Sciences, or Engineering

1

2. Sociologist/ Economist

3+ years in capturing socio-economic aspects of development interventions

Masters level — Social Sciences (economics/sociology or other similar disciplines)

1

3. Engineer 3+ years in field Bachelors level —Engineering Electronics / Telecom

Workload dependent

PROCUREMENT DOCUMENTATION

Part 2, Session 5

List of Important Documents for Pre-qualifications

1.  Invitation to Prequalify

2.  Instructions to Applicants

3.  Prequalification Datasheet

4.  Prequalification Evaluation Criteria

5.  Prequalification Application Form.

Major parts of RfP for M&E

1.  Introduction

2.  General Instructions to Bidders

3.  Specific Instructions for Technical & Financial parts of Bids

4.  Evaluation Criteria (within the bidding docs!)

5.  Draft Contract

Structure of TORs

•  Preamble

•  Brief Project Description

•  Scope of Work

•  Key Personnel

•  Obligations of Successful Bidder

•  Time Frame

•  Deliverables

•  Payment Modalities

THANKS

[email protected]

Backup Slides

Agenda Part-1

•  Session 1- Setting the stage •  Session 2- Monitoring •  Session 3- Evaluation

TORs for the Evaluation Team/Firm

•  Evaluation Background and Context •  Evaluation Purpose •  Evaluation Scope and Objective

–  What aspects of the intervention are to be covered by evaluation? (time frame, geographic area & target groups to be considered, projects (outputs) are to be included).

–  What are primary issues of concern that evaluation needs to address ?

•  Evaluation Questions •  Evaluation Methodology

…continued

TORs for the Evaluation Team/Firm

•  Evaluation Products –  Inception report

–  Draft Evaluation report

–  Final Evaluation Report

•  Evaluation Team Composition & Competencies

•  Evaluation Implementation Arrangement

•  Evaluation Time Frame for the Process

•  Evaluation Cost – Projected Budget

Planning

Goal Setting

Developing Strategy

Resource Allocation

Outlining Implementation

Sourcing Options

Steps for M&E implementation in the 3 scenarios

Scenario 1 Scenario 2 Scenario 3

Recom

mendations for activating M

&E Function

Implement M&E in-house •  Execute

M&E Plan

Implement M&E in-house, but: •  Acquire deficient HR •  Develop M&E team •  Develop capacity of team thru

professional training •  Procure necessary HW & SW •  Develop M&E Plan •  Execute M&E Plan •  Outsource functions which are

not feasible or are occasional

Outsource M&E •  Engage a firm

for provision of M&E services

Possible reasons of selecting one over the other

In-House

•  Cost - may be less expensive in long run

•  Required expertise is available

•  Want direct control over the process

•  Desire of better quality

•  Flexibility in managing system configuration

•  Reliable & competent service providers not available

•  Routine nature of task

Outsourced

•  Cost - presently less expensive to outsource (eg: there isn’t lot of work)

•  Lack of in-house capacity very big

•  Outside Service Providers’ research & specialized know-how exceeds

•  External validation is required (like third party audit validation)

•  To keep the in-house manpower small

•  May be a Legal requirement

•  Notion that third-party M&E is considered more “neutral” than in-house.

Structure of a typical M&E Unit

Material Resources Required for In-house M&E

•  Hardware •  Software •  Drive test equipments •  Furniture •  Communication Equipment •  Vehicles •  Office

Selection Process to outsource M&E

WORKED EXAMPLE FOR EVALUATION OF BIDS

Part2, Session 6

Worked example - Evaluation of Bids

Items Assigned weights

Evaluated Rating Points

1- Qualifications of Bidder (150)

a- Experience in similar projects 50 100% 50

b- Experience in similar geographic areas 50 90% 45

c- Prior experience of working with USPF 50 0% 0

2- Approach and methodology (400)

a- Understanding of objectives 100 90% 90

b- Quality of methodology 100 100% 100

c- Innovation in proposed approach 50 90% 45

d- Work plan/ time lines 50 80% 40

e- Presentation of the Bid 50 90% 45

f- Partnership with a local firm 50 70% 35

(…cont)Worked example - Evaluation of Bids

Items Assigned weights

Evaluated Rating Points

3- Personnel (450)

a- Team leader 100 80% 80

b- Monitoring specialist 50 70% 35

c- Evaluation specialist 50 90% 45

d- Survey specialist 50 100% 50

e- Sociologist/Economist 50 80% 40

f- Reporting Specialist 50 90% 45

g- Other Staff 100 90% 90

Total Evaluated Points 835