Integrated Assessment Approach M&E and IE
Feb 22, 2016
Integrated Assessment ApproachM&E and IE
6 Purposes of Impact Planning, Assessment, Reporting & Learning (IPARL)
• Improve projects
• Strengthen relationships
• Demonstrate impact
• Inform strategy
• Sustain credibility and legitimacy
• Educate society
Integrated Assessment ApproachM&E and IE
• Baseline Data/Evidence
• KPI Scorecards
• Evidence of Change Journals
• Comparative Constituency Feedback (CCF)
• Rapid Advocacy Learning Activity Reports
• Comparative Case Studies
KPIs Scorecards
Evidence of Change Journals
• Constituents’ perceptions of CEPA’s performance and impact
• Quality of relationships within CEPA
• Comparative analysis
• New insight on the campaign’s effects• Information on key relationships• Inform assessments and course correct• Amplify voice and participation• Legitimacy• Advocacy tool
National level Global level
• Partners• Allies• Media
• UNAIDS• PEPFAR• Global Fund• UNITAID• CHAI• Unicef
Relationships Impact
• To what extent has your association with CEPA met your expectations?
• How would you rate the quality of the communications within CEPA?
• How would you rate CEPA’s impact in the field of paediatric HIV/AIDS?
• In which ways is CEPA contributing to the elimination of paediatric HIV/AIDS in your country/globally?
Oct-Dec 2009 Design and implementation of global level GAA/CEPA baseline survey
Oct-Dec 2009 Design of country level CEPA baseline survey
Jan- March 2010 Implementation of the country level baseline survey
Q4 2010 Implementation of global and country level constituency surveys
Q1 2012 Repetition of global and country level constituency surveys
Rapid Advocacy Learning Tools
• Bellwether Methodology
• Policymaker Ratings
• Advocacy Pre-mortem
• Intense Period Debriefs
Rapid Advocacy Learning Tools
Intense Period Debriefs
• Occur shortly after policy windows or intense activity periods• Gathers in-depth and real-time information
• Convening of either a focus group or the conducting of individual interviews with advocacy stakeholders to capture data about advocates’ recent experiences.
• Captures:• The public mood and political context during the policy window;• What happened and how the campaign members responded to eve
nts, especially as related to actions that occurred behind closed door s;
• Perspective on the outcome(s) achieved or not achieved;• How strategies might be adjusted in hindsight.
Intense Period Debriefs
Sample Questions1. What events triggered this intense period? 2. How was the organization’s response determined? Who w
as responsible for that decision? How was that decision co mmunicated to partners and allies?
3. Which elements of the organization’s response worked well ? Which elements could have been improved?
4. What was the outcome of the intense period? Was the resu lt positive or negative?
5. What insights will you take away from this experience that might inform your strategies going forward?
Comparison and Triangulation “A Platinum Standard of Rigor”
For Impact Evaluation
Comparison and Triangulation of:
• Alternate theories of action (casual pathways)• Before and after, counterfactual• Contexts (location, time, scale)• Multiple types of information and data• (Competing) stakeholder perspectives
Rigorous Comparison and Triangulation Advances in Case Study Designs
Case studies still a central “method” and can be made more rigorous with relatively small investments
Case studies essential for analysis of:• context – location, time, scale, scope• descriptive inference: construct validity• causal processes: internal and external validity
(simple, complex, complicated, equifinality, etc..)• multiple results (outputs, outcomes, impacts)• unintended results/unexpected consequences• triangulate qualitative and quantitative data• useful communication and reporting tool
Rigorous Comparison and Triangulation Advances in Case Study Designs
Purposes and types of Case Studies:
• Generating theory of change – “inductive”• Elaborating TOC – “plausibility probes”• Evaluating TOC – “least likely” and “most likely” “tough
tests”, “counterfactual thought experiments”• Innovating theory of change – “deviant case studies” of
exceptional “outliers”
Rigorous Comparison and Triangulation Advances in Case Study Designs
Case Study Methods and Tools:• Process-Tracing• Analytic Induction• Structured-focused comparison• General (and Specific) Elimination• Typological Theorizing• Increasing Observations• Data-bases of Case Studies
… and so many more…