Evaluating networks slides_final_monitor3.8.11

Innovations for Scaling

Impact

Monitoring and Evaluating Networks

March 5, 2011


Impact

Seminar Overview

Overview of Networks as Complicated and Complex Initiatives - Network Life-Cycle

Network Theories of Change and Action

Developing a Network Integrated Assessment Framework – Focus on Network Metrics

Learning and Reporting Activities for Continuous Improvement

Overview of IPARL system Across the Network Life-Cycle


Impact

Impacts Community of Practice

Initial focus on sharing and developing innovative approaches to monitoring and evaluating networks and network approaches – now moving to greater focus on advocacy

Includes – Evaluation experts, practitioners, network representatives and funders (in-person and virtual interactions – www.scalingimpact.net)


Impact

Additional Materials

Network Briefs

Next Generation Network Evaluation Paper

Step-by-Step guide to developing Comparative Constituency Feed Back Surveys for Networks

Available at www.scalingimpact.net


Impact

Introduction and Context

Networks and networking have become increasingly and consciously utilized

Increased interest and investment in networks from a range of actors

Growing demand for more success from public and private initiatives has created explosion of interest for increased and improved M&E/IE

Together these trends have created a growing appetite for Network M&E/IE


Impact

Current Field of Network Monitoring and Evaluation

Still in its infancy in terms of theory and even more so in terms of practice

Most M&E work using network methodologies to date have been small in size of networks and number of participants

Number of metrics, tools and broader approaches developed specifically for networks is growing but not many practical tests/examples

Much can be borrowed from other fields like complexity science and systems thinking


Impact

Gaps in Current Practice

Growing number of network metrics but not utilized in practice

Most tools and approaches only address one or a few of the specific challenges faced by networks

Little explicit focus on revising and refining network’s ToC

Few comprehensive frameworks for developing holistic M&E systems

Those that do encourage development of broader M&E systems focus more on planning than M&E


Impact

Networks as Complicated and Complex Initiatives

Networks and networking strategies:

Involve multiple factors, actors and contexts

Are non-linear, path dependent and emergent

Network implementation varies even when replication mechanisms are applied because of contextual differences


Impact

Standardized, Complicated and Complex Initiatives


Impact

Network Life Cycle

(Simplified)

•Increasing and diversifying network membership and participation•Developing network structures (geographically and/or functionally)•Spreading and deepening initial key strategies/diversifying/pruning strategies• Business model development, diversification and sustainability•Enhancing legitimacy and value•Building enduring social connectivity/trust•Monitoring, learning, continuous improvement and course corrections•Implementing a robust network communication platform

Enhancing and Expanding

•Agreeing on shared norms and principles•Establishing initial network structures and processes (governance, admin, legal form?)•Piloting key change strategies and tactics•Formulating network theory of change and participant theories of action•Designing a user centered network communication platform •Business model formulation

Launching

•Mapping of issue/problem and key stakeholders, roles and relationships•Convening leadership group, key stakeholders and relationship building•Defining the initial vision, purposes and value proposition• Securing seed funding

•Mapping of issue/problem and key stakeholders, roles and relationships•Convening leadership group, key stakeholders and relationship building•Defining the initial vision, purposes and value proposition• Securing seed funding

Catalyzing

•Evaluating network vibrancy, effectiveness and sustainability •Deciding to transform or transition•If transforming-

•Refining/redefining network vision, purpose and value•Redirecting the issue/problem system and altering core relationships

•If transitioning-•Potential evolution to non-network structures and processes•Distribution and utilization of re-usable assets

Transforming or Transitioning


Impact

Network Assessment in Summary

Networks are Complicated Understand both the network “parts” and “whole” Develop appropriate network metrics Gather comparable and consistent data across the

network

Networks are Complex Identify impacts Utilize results for real time learning and

improvement Manage emergent outcomes

Life cycle approach is really helpful in monitoring and evaluation of networks


Impact

Group Discussion

In what ways do networks achieve impact/What are various network “impact pathways”?

What types of theory of change approaches are useful for networks?


Impact

Describe Network Impact Pathways

What are the various impact pathways the network and its members will use?

Will the network act as a: Filter Amplifier Convener Community Builder Funder Capacity Building Facilitator

Most networks will use multiple impact pathways


Impact

Network Theories of Change and Action

Identifies desirable results (What)

Articulates the multiple and often interacting pathways for affecting change (How)

Maps out the actors that contribute to various pathways (Who)


Impact

Tools You Can Use

Outcome Mapping

Causal Mapping

Systems Approaches


Impact

Steps to Developing a Network ToC

Articulate the long-term network goal Describe the various impact pathways

the network and its members will utilize

Describe the medium and short-term outcomes and outputs the network can affect through its various strategies that will contribute toward the long-term goal


Impact

Articulate Network Long-Term Goals

What is the network’s long-term goal/vision?

Opportunity to: review and confirm original long-

term goal with members Or…..

revise and refine original long-term goal with members


Impact

Example

Global Knowledge Partnership - a network of public, civil society and commercial organizations, works to provide members with access to global knowledge and innovation

Filters and disseminates knowledge and innovations gathered from its members and others throughout its network

Convenes members and others to share good practices and new knowledge during conferences and smaller workshops

Provides grants to members to support their activities

Facilitates learning across members through member meetings and joint projects


Impact

Example

Campaign to End Pediatric HIV/AIDS, which focuses on advocacy as a means to increase coverage rates for pediatric HIV/AIDS across six partner countries:

Amplifies a particular set of ideas and policy stances on pediatric HIV/AIDS

Convenes partners at the global, national and regional level across six partner countries in order to organize a cohesive advocacy campaign

Provides resources to each of the official country partners to organize local advocacy efforts


Impact

Describe Network Outcomes

What are the network’s medium and short-term outcomes that contribute toward the long-term goal?

1.Brain-storm preconditions for success

2.Group outcomes into broader categories

3.Identify the specific outcome categories that the network will focus on


Impact

Steps to Developing Network Theories of Action

Examine how (or how not) the members goals align with the network’s long-term goal

Describe the specific member’s impact

pathways that will contribute toward the network’s long-term goal

Describe the medium and short-term

outcomes and outputs the individual member can affect through its various impact pathways that will contribute toward the networks outcomes and long-term goal


Impact

Examine Member and Network Goal Alignment

How (or how not) do an individual member’s goals align with the network’s long-term goal?

Each member examines: 1. How their goals relate to those of the

network2. How/where their activities can best

support the network


Impact

Describe Member Impact Pathways

What are the specific pathways a member will use to support the network’s long-term goal?

Each member:1. Reviews the original list drafted by the

network2. Identifies which pathways they will

focus on – those that best support their own and the network’s goals


Impact

Describe Member Outcomes

What are the short and medium-term outcomes the individual member can affect that contribute toward the network’s outcomes and long-term goal?

Each member Describes the outcomes they will work toward Identifies how these relate to those of the

network

Note: one member level outcome may contribute toward multiple network level outcomes


Impact

Describe Member Outputs

What are the short-term steps that contribute towards and are necessary for the achievement of member outcomes?

Each Member Describes outputs Identifies how they contribute toward

their outcomes


Impact

Group Discussion

What types of metrics are particularly applicable for networks?

What types of assessment tools are particularly useful for networks?


Impact

Developing a Network Integrated Assessment

Framework Links ongoing monitoring and

evaluation with punctuated outcome and impact evaluation

Includes: A set of network metrics for monitoring

progress tied to the identified outcomes A set of network tools to support the

collection of metrics Periodic evaluation including impact

evaluation


Impact

Steps for Developing a Network Integrated Assessment

Framework Gather information about members’ current M&E

practices and capacity

Develop Measures Identify output and outcome indicators Identify vibrancy and connectivity indicators

Identify the tools that can be used to support data collection

Develop the specific process (who will collect the data, how often, etc…) for collecting and storing data

Develop and initial time-line including a schedule for periodic evaluation

Develop a system for collecting and recording data


Impact

Developing Measures

Overall network effectiveness is combination of: Network Vibrancy Network Connectivity Network Effects

Specific indicator can be developed for each metric

Different categories of metrics may be more or less important depending on context


Impact

Example Network Metrics and Survey/Stakeholder Interview

QuestionsNetwork Vibrancy

Alignment Is there a match between member’s goals/priorities and that of the network?

Diversity How diverse is the network?Is the current level of diversity appropriate for the network’s goals?

Network Connectivity Participation How active are members?

What is the value of participating in the network?Communication How well do the various network parts communicate

with each other?How often do network members communicate

Network EffectsInfluence How influential is the network?

Value added Is the network fully realizing the advantages of working together?


Impact

Identifying Output and Outcome (Effects) Indicators

What are the network and its members achieving?

What are the indicators that can be used to track progress towards outputs and outcomes?

Output and outcome indicators should be tied to outputs and outcomes identified in the ToC/ToA


Impact

Identifying Vibrancy and Connectivity Indicators

How well is the “network” itself functioning?

What value added does using a network approach bring?

Network and members should monitor at least some components within the vibrancy and connectivity categories

Vibrancy and Connectivity indicators can help test original assumptions about network approach


Impact

Selecting Network Tools

Tools can be used in combination to support data gathering for specific metrics of interest

Identify the indicators and then select the appropriate tool to help gather data


Impact

Example Vibrancy Tools

Tool Purpose Strength

Comparative Constituency Feedback Surveys

Gather stakeholder perceptions that can then be compared across similar networks

Allows for benchmarking and can foster learning across networks. Opportunity to incorporate stakeholder perceptions.

Network Effectiveness Diagnostic Tool

Short tool to identify the strengths and weaknesses and to explore possible actions to develop or strengthen the network

Relatively simple activity requiring limited time and resources. Responses could be compared across stakeholder groups.

Partnership Score Card

Identifies where partnerships are excelling, where they are breaking down and why. Assesses partnerships by analyzing the tangible and intangible exchanges between the parties..

Provides detail on specific partnerships of interest. Performance targets can be set and measured/assessed at regular intervals


Impact

Example Connectivity Tools


Social Network Analysis

Examines the links, relationships and behaviors between actorsAnalyzes behavior patterns and group structure - Can be used to asses changes in network structure over time

Good for understanding the relationships between actors within the network and overall network structure

Value Network Analysis

Maps activities and sets of relationships identifying both tangible and intangible transactions and roles within the network

Examines what specifically is being “exchanged” within the network . Identifies both tangible and intangible value.

Monitoring at the edges

Monitor levels of networking that do not directly involve the secretariat

Relatively simple activity requiring limited time and resources


Impact

Example Social Network Analysis Map


Impact

Example Effects Tools


Strategic Leverage Index

Examines how groups come together to achieve their own and broader collective goals - Creates a conversation around valuesWorks to measure stakeholders inter-dependencies

Creates a picture of network “health” across stakeholder groups and levels

Impact Pathway Evaluation

Impact pathway (or ToC) developed at the start of the initiative -Monitoring of indicators aligned withthe ToC used to guide the initiative and revise/refine the ToC in real time - Evaluation seeks to establish plausible links between the initiative’s ToC and subsequent developmental changes

Links external punctuated evaluation to continuous internal M&E - Makes explicit, tests and refines an initiative’s ToC

Contribution Assessment

Understand how and where members are interacting with the network and what their priorities are

Relatively simple participatory activity requiring limited time and resources


Impact

Collecting and Recording Data

After indicators and tools have been selected, members should develop an initial time line including: Identifying who will be responsible for collecting data Identifying who will be responsible for recording data Determining how often data will be collected Determining where and how data will be stored Determining how often punctuated evaluation will be

conducted

Data can be stored in a variety of formats (tables, databases, online, etc…). members, stakeholder should discuss and decide: Who will be the primary users of the data? How often will data be recorded? What needs to be recorded (qualitative or

quantitative information)


Impact

Group Discussion

Is it possible to conduct an impact evaluation of networks?

How can impact evaluations of networks be rigorously implemented?


Impact

Impact Evaluation of NetworksSome Assumptions and

Terminology Bayesian approach to evaluation

Evidence requires theory (interpretation)

Match design to stage of theory development Approaches/methods vs data collection tools

Quantitative and qualitative (data)

Data-set vs. causal process observations


Impact

Network Impact Evaluation

Map out Causal Chains Anticipate Heterogeneity (Distributional

Impacts) Anticipate Unintended Consequences Understand Context Rigorous Factual Analysis Rigorous Counterfactual Analysis Use Mixed Methods

Theory of Change is Touchstone


Impact

Comparison and Triangulation:

Within and Across Evaluation Designs

Ethnographic

Single case study

Comparative (case-study)

Statistical/econometric

Experimental/quasi-experimental


Impact

The Platinum StandardThe Platinum StandardComparison and Comparison and

Triangulation to Address Triangulation to Address Threats to Validity Threats to Validity

Construct Validity - how closely a measure reflects the underlying concept of interest

Statistical Conclusion Validity - whether, and by how much, presumed causes and effects co-vary

Internal Validity - whether observed co-variation represents an actual causal relationship within a specific context

External Validity - the extent that a causal inference drawn from one or more contexts/cases applies to other contexts/cases

Trade-offs across all evaluation strategies


Impact

Rigorous Advances in Comparative Method/Case

Study Designs Comparative and Case Study Methods/Tools: Process-Tracing Analytic Induction General (and Specific) Elimination Counterfactual Thought Experiments Structured-focused comparisons Boolean Algebra Data-bases of Case Studies

… and so many more…


Impact

Rigorous Advances in Comparative Method and Case Study

Designs Case studies still a central “method” and can be

made more rigorous with relatively small investments

Case studies essential for analysis of:- context – location, time, scale, scope- descriptive inference: construct validity- causal processes (simple, complex, complicated)- multiple results (outputs, outcomes, impacts)- unintended results/unexpected consequences- triangulate qualitative and quantitative data- useful communication and reporting tool


Impact

Rigorous Advances in Comparative Case Study

Designs

What is this a Case of?

With-In Case Analysis

Cross-Case Analysis


Impact

Rigorous Comparison and Triangulation Advances in Case

Study Designs

Purposes and types of Case Studies: Generating theory of change – “inductive”

Elaborating TOC – “plausability probes”

Evaluating TOC – “least likely” and “most likely” “tough tests”, “counterfactual thought experiments”

Innovating theory of change – “deviant case studies” of exceptional “outliers”


Impact

Learning and Reporting Activities

for Continuous Improvement What are the unique types of

learning and reporting needs/goals for continuous improvement of networks?

What types of learning and reporting tools are particularly useful for networks?


Impact

Continuous Learning and Improvement Activities

Structured activities that create space to analyze results in order for the network and its partners to learn and course-correct in real time

Lessons learned shared across partners at both the global, regional and national levels to scale-up success and overcome challenges

Activities are used as a planning input to future programs and investment as a way to maximize impact

Example activities include: Interrogation of monitoring reports, scorecards, and

data Stakeholder Ratings Intense Period Debriefs


Impact

Example Learning Activity: Comparative Constituency

Feedback Survey Structured questionnaire to collect

comparative feedback from a network’s key constituents

Steps: Setting up a cohort – within or across networks Setting up an Advisory Group - Designing the questionnaire Administering the survey Producing comparative reports Follow up


Impact

Example Learning Activity: Intense Period Debrief

Structured reflection activity after intense period of activity

Steps: Identify an “intense period” that you want to

reflect upon as a team Designate someone to facilitate Either in focus group or individually,

facilitator asks participants a set of questions Network members to discuss results of the

debrief, what has been learned, and what might be improved for future network efforts

Record results of debrief in the evidence of change journal


Impact

Example Intense Period Debrief Questions

What events triggered this intense period?

Which elements of the organization’s response worked well? Which elements could have been improved?

What was the outcome of the intense period? Was the result positive or negative?

What insights will you take away from this experience that might inform your strategies going forward?


Impact

Developing Stakeholder, Public and Donor Reporting Processes

Explicitly sharing evaluation findings with constituents:

Provides new insight about effects on key constituents, can inform the network’s assessments of its efforts, and help it to course correct.

Informs the network’s understanding of important, new aspects of its performance.

Empowers constituents by amplifying their voices. The process gives constituents a new opportunity to participate in the network and increase their sense of stake in it.

Helps the network grow in legitimacy as a result of its visible efforts to be accountable to its constituents.


Impact

What should be included in a public report?

Formats may vary, but public reports should address questions like: What was to be achieved? What was achieved? What caused or hindered success? What’s working well? What should be changed and why? What has been changed and how?


Impact

Network Impact Planning, Assessment, Learning and

Reporting Systems Overview

An overall framework for planning, monitoring and evaluating networks

Consists of a general set of key components and a menu of tailored options

Meets some of the challenges of network evaluation


Impact

Key Network IPARL Components

A clearly articulated theory of change and aligned theories of action

An integrated assessment framework composed of: A set of network metrics for monitoring progress and

accountability A set of network tools Periodic evaluation – Including impact evaluation

Stakeholder, public and donor reporting and feedback

Continuous learning and improvement mechanisms


Impact

Group Discussion

Which network monitoring, evaluation and learning ideas and tools seem most helpful and why?

How might I use different network monitoring, evaluation and learning tools?

How might I use an IPARL system in my work with networks?


Impact

Additional Materials

Network Briefs

Next Generation Network Evaluation Paper

Step-by-Step guide to developing Comparative Constituency Feed Back Surveys for Networks

Available at www.scalingimpact.net

Evaluating networks slides_final_monitor3.8.11

Documents

term goal

continuous

term goal

key stakeholders

network life

connectivity

impact pathways

reporting