1 Workshop Monitoring a evaluace lidskoprávních projektů Monitoring and evaluation of human rights projects Inka Píbilová inka @ evaluace.com 3. prosince 2013
May 11, 2015
1
Workshop Monitoring a evaluace lidskoprávních projektů
Monitoring and evaluation of human rights projects
Inka Píbilová [email protected]
3. prosince 2013
2
Monitoring x evaluation x audit
Evaluation • Assessment of project efficiency, effectiveness, impact, relevance and sustainability for the purpose of learning and accountability to stakeholders
Monitoring • Ongoing analysis of project progress towards achieving planned results with the purpose of improving management decision making
Audit • Assessment of (i) the legality and regularity of project expenditure and income i.e. compliance with laws and regulations and with applicable contractual rules and criteria; (ii) whether project funds have been used efficiently and economically i.e. in accordance with sound financial management;; and (iii) whether project funds have been used effectively i.e. for purposes intended.• Primarily a financial and financial management focus, with the focus of effectiveness being on project results.
http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm
3
Why to monitor and evaluate projects?
• Accountability
• Learning
4
Project cycle management
Programming
Identification
FormulationFinancing
Implementation
Evaluation
http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm
5
Stakeholder Analysis Matrix
Example: River Pollution
http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm
6
Stakeholder Analysis – Power / Interest / Attitude In
fluen
ce/P
ower
KEEP SATISFIED INFLUENCE & ENGAGE
KEEP INFORMEDMONITOR
Interest
Policy makers
CSOsActivists
Attitude
7
Problem Tree
Example: River Pollution
http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm
8
Objectives and strategy selection
Example: River Pollution
http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm
9
Inputs – Activities - Outputs – Outcomes – Impacts
http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm
Direct influence of implementers:Inputs I need for my work… (funds, people, time)Activities I do….
(research, conference)Outputs I produce …
(research paper, 100
participants to
conference)
Indirect influence of implementers:Outcomes I strive to…(Results) (raise awareness on X)Impacts I aim to…
(reduce human rights
violations)
10
Logical framework
http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm
11
Logical framework
http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm
Issues identified during project evaluations:• Logical frameworks too complicated, too many objectives / indicators• Too ambitious and/or too vague objectives• Indicators difficult to measure / data not collected or difficult to collect• Logframe not used for internal monitoring / evaluation and decision
making
Tips:• Numbering Objectives (1,2), Results (1.1., 1.2., ...), Activities (1.1.1.,
1.1.2.)• Check source of verification – how will data be collected, by whom,
when/how regularly?
12
Quantitative - SMARTERSpecific / Simple (to understand, collect)MeasurableAttainable/Available at cceptable costsRelevant to project / stakeholdersTime-boundEvaluate/EngagingReevaluate/Recordable
Use both quantitative and qualitative indicators
http://www.smarttoolkit.net/?q=node/391http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm
Qualitative - SPICEDSubjectiveParticipatoryInterpreted and communicableCross-checked and comparedEmpoweringDiverse / disaggregated (by gender)
Compare using trends (increase), thresholds (min. 30%), targets (strategy by 12/Y1)
Min. 30 % of participants initiate a project aiming to address a local issue.
Reasons why participants have (not) implemented a project to address a local issue
13
Risks Analysis
http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm
Assump-tions in
logframe
14
Monitoring: Why, for whom, what, how
Why?• Timely identification of
successes and problems during project implementation
• Informed and timely decision making by project managers to support implementation
• Accountability for the resources used and results achieved
• Stakeholder awareness and participation
• The evaluation of project achievements and audit of activities and finances
15
Use logframe to prepare Monitoring Plans
Year 1Indicator / Country
CZ PL SK Total Note
15.000 copies of publications distributed (5.000 each)
90% 110% 100% 100% Achieved
120 teachers trained (40 each) 45 40 35 120 AchievedMin. 50% of teachers implement the new tools within 6 months
60% 40% 30% 40% Not achieved
• Use indicators from the logframe• Split by year and partner• Use plan vs. actual, % or trends over time• Develop adequate sources of verification• Distribution lists• Attendance sheets with contacts!• On-line feedback forms after 6 months (combined with
supervision of teachers)• Media monitoring, public surveys...
16
Monitoring: Activity schedule with tasks and responsibilities
17
Monitoring Progress Report (Narrative)
http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm
Monitoring Visit Checklist
18
Other monitoring tools
Forms at docs.google.com Geolocation at www.googlemaps.com, www.ushahidi.com
Monitoring tips
• Keep the users of information clearly in mind (who needs what information?)
• Build on local information systems and sources - costs and sustainability of new systems
• Collect only the minimum amount of information required – keep it simple and practical, more information is not better information
• Use Inception reports when team differs from the proposal writters, or when time passed by between identification and grants approval... usually 3 months after project start, revises original project proposal - updates logframe, sets annual action plan and monitoring plan
20
Evaluation: Why, for whom, what, how
21
Plan external evaluation well aheadTe
rms
of R
efer
ence
– O
bjec
tives
, sc
ope,
stak
ehol
ders
, que
stion
s,
budg
et, s
ched
ule,
out
puts
, use
.
Desk study Interviews
Surveys
Focus groups
Case studies Prel
imin
ary
findi
ngs
&
conc
lusi
ons
Finaldebriefing of all partners
Communication with the Project Partners
Draft evaluation report commented by all partners Fi
nal e
valu
ation
repo
rt
Inception phase
1-3 months
Field research
1-3 months
Reporting phase
1-2 months
Initi
al b
riefin
g an
d in
cepti
on
Evaluator Selection
22
Evaluation criteria
23
EC Evaluation Criteria
24
How to measure learning outcomes?
http://leanlearning.wikispaces.com/learning_analytics
25
Evaluation tools
26
What does participation mean?
27
Reporting: Why, for whom, what, how
28
Different evaluation methods and approacheshttp://betterevaluation.org/
Where to learn more
Road to Resultshttps://openknowledge.worldbank.org/bitstream/handle/10986/2699/52678.pdf?sequence=1
Evaluations of Czech development and humanitarian projectshttp://www.mzv.cz/jnp/cz/zahranicni_vztahy/rozvojova_spoluprace/dvoustranna_zrs_cr/evaluace/index.html
EPDET - annual training in Evaluations by World Bank consultantshttp://www.dww.cz/english.php?page=epdet1
EC Project Cycle Management Manualhttp://www.europa.eu.int/comm/europeaid/qsm/index_en.htm