InterAction - Rockefeller - TCC - GlobalGiving

Advances in Development Evaluation

InterAction Forum 2012

Session Overview

• Interactive - audience participation • Global Giving’s storytelling work• Evaluation of networks• Collaborative learning activity• Wrap up / Q&A

Introduction

• The Rockefeller Foundation Evaluation Office is supporting innovative evaluation approaches, new methods and tools and evaluation capacity development

• This session presents the work of two of RF’s grantees who are key contributors to advances in development evaluation

SKILLS

IDEASTHE EXPERIENCE

Tell us about something that surprised you at InterAction yesterday.

What is your story about? Place it somewhere inside the triangle, or in the “other” box.

OTHER

GlobalGiving Last Week

Catalyze Marketplace of $, Info, Ideas

Accessible to Growing # of More Effective Orgs

Democratizes Aid and Philanthropy

GlobalGiving Impact Framework

Design Model: Yelp! Changes the Market

Feedback makes it safe (and satisfying) to give to riskier (but maybe better performing) orgs.

Feedback identifies great NGOs and helps them learn.

Resulting data is useful for other funders, NGOs, government agencies, and researchers.

Closing the Broken Feedback Loop

Methodological Logic

Story Capture

Self-Signification

Visualization (Theme, Geo, Org, Time)Discussion

Act on InsightsMany

Narratives

Contextualized Statistics

Emergence

What’s working?

Working! • Real-Time Decision-Making Aid for GlobalGiving• Story Patterns Illuminate Needs• Actionable Constituent Voice

Promising!• Performance Dashboard for NGOs• Methodology Recognized as Useful• Interoperability with other Monitoring Tools

Working on it!• Globally Scalable Collection Infrastructure• Market-Driven Incentives for Story Collection• Open-Source Analysis Tools

strategies to achieve social impact

Jared RaynorDirector of Evaluation

InterAction ConferenceMay 2012

Evaluating Coalitions and Networks

Strategies: Outcomes: Impact:Project Inputs/ Resources:

Logic Model: Advocacy Initiatives

Issue analysis/ research

Media advocacy/ Public awareness

raising

Grass roots organizing

Coalition building/ networking

Policy analysis/ research

Legal action

Assure good policy implementation (Administrative/

Regulatory oversight, TA,

monitoring, etc.)

Lobbying and direct policy-maker

influence

Imp

rov

em

en

t in

th

e Q

ua

lity

of

Liv

ing

fo

r th

e C

om

mu

nit

y

Better defined and framed problems

Policy decisions effectively implemented

Policy makers adopt new policies based on the agenda / do not adopt harmful policies

Issues more clearly set on the public agenda and gain prominence (momentum, interest, awareness, etc.)

Impact of new policies is evaluated

Advocacy Staff: • Time, experience and

expertise (administrative, legislative, election-related and legal)

• Core skills (analytical, communication, research, etc.)

Funding:• Monetary resources• Non-monetary/ non-

staff resources (e.g. volunteers, in-kind donations)

Organizational Reputation:• Advocacy related• General perception as

a quality organization• Representative for

base constituency

Network:• Policy makers• Partner organizations• Media• Mobilized base

(C) TCC Group 2010

Ch

an

ge

in S

oci

al S

tru

ctu

re R

efle

ctin

g P

osi

tive

So

cia

l Ju

stic

e S

hift

Co

ali

tio

n/N

etw

ork

(a

s a

n A

cto

r a

s co

mp

are

d t

o a

st

rate

gy)

No

np

rofi

t o

r A

dv

oc

ac

y

Org

an

iza

tio

n (

use

s co

alit

ion

as

a s

tra

teg

y)Acting

Organization:

Continuum of Inter-Organizational Relationships (IORs)

Mutual Accountability

SeparateAccountability

InformalOrganization

Formal Organization

Informal Network

Coalition

Collaboration

Strategic Alliance

Joint Venture

Merger

Committee

Formal Network

Adapted from: Wolf, T. (2003). A Practical Approach to Evaluation of Collaborations. In T.E. Backer (Ed.). Evaluating Community Collaborations. New York, NY. Springer Publishing Co. with additional acknowledgement of K. Hobson and M. Hightower King at AEA 2009.

Relationships for What Purpose?

• Respond to complexity• Cope with turbulence/complexity• Acquire resources (while maintaining autonomy)• Decrease transaction costs• Gain legitimacy/power

Positive

Externalities(Value

)

Negative

Externalities (Congestion)

What Are We Measuring?

Coalition

Individual

An Entanglement of Relations

Coalition

Individual

Organization

Individual

Organization

Individual

Organization

Individual

Number of Relationships

2 Person Coalition: 83 Person Coalition: 184 Person Coalition: 32

A Simplified Approach for Evaluation

Member Capacity

Network/ Coalition Capacity

Outcomes

Image credit: http://www.cugelman.com/research/united-nations-web-network.htm

Member Capacity

Member Capacity

• What do you want to get out of the network/coalition? • How do you justify your involvement?• How much time can you devote?• What can you offer?

Individual Capacity

Network Coalition

Boundary spanners Skill/knowledge to work collaboratively

Time/space to engage in network Time and Commitment to working in coalition

Relevant skills and interest Relevant decision-making authority

Desire to be part of something bigger

Strategic use of coalitions to fill critical gaps and leverage resources toward achieving your mission

Coalition/Network Capacity:A Framework

The Critical Organizational Capacities for Advocacy Initiatives: The Logic Model “Inputs” Initiative Logic Model

Imp

rove

men

t in

th

e Q

ual

ity

of

Liv

ing

fo

r th

e C

om

mu

nit

y

Issue analysis/ research

Media advocacy/ Public awareness raising

Grass roots organizing

Coalition building/ networking

Policy analysis/ research

Legal action

Administrative/ Regulatory implementation influence

Lobbying and direct policy-maker influence

Management:

Adaptability:

Technical:

Access window of

opportunity/ threat:

•Offense•Defense

Org. Culture

(C) TCC Group 2008

Leadership:

A Few Key IOR Capacities

Goal Destination

(What is the IOR trying to achieve?)

Value Proposition

(Why is this IOR the right approach?)

• Rules, procedures and decision-making• Action/Exchange-oriented• Leadership

Entropy

(disorder)

Bureaucrac

y(rigidit

y)

Multiple Levels of Outcome

Community

Individual

Benefits to Individuals

Community Impact/

OutcomesDevelopment

of IOR

Benefits to Organizations

Capturing ‘Externalities’

• Better implementation/coordination of actual programs addressing the issue (More entrenched programming)

• Cross-fertilization and Innovative ideas (Group think)

• Other specific project goals met

• Sustained networks/relationships (Relationships hurt)

• Reduced sense of isolation (Personality conflicts)

Outcome questions for IORs in Emergency Settings

• Has the IOR increased the ability to coordinate during varying stages of the emergency, including preparation?

• Did the IOR reduce political complexity?

• Did the IOR reduce or increase time and resource constraints?

• Has the IOR increased the reliability of information available?

• Has the IOR facilitated overcoming infrastructure challenges?

• Has the IOR enhanced the well-being of personnel?

• Has the IOR facilitated better relationships with affected populations and donors?

• Has the IOR increased ability to engage with the media?

Conclusion

Contact Information

Jared RaynorDirector of Evaluation

TCC Groupwww.tccgrp.com

Leveraging the Evaluator

• Navigate power differentials• Use to:

• Build (inform development) • Sustain (strengthen existing

performance; identify inefficiencies)

• Reflect (accountability for resources and prep for future)

• Understand how to position yourself in a network

• Help articulate goal destination and value proposition

Mobile Phone Principles of Good IOR Behavior

Getting the Right Plan

• What do you need to accomplish?

• Do you have appropriate bandwidth (value vs. congestion)?

• Right amount of minutes/airtime?

1

Mobile Phone Principles of Good IOR Behavior

Aware of Surroundings• How should others contact

you? You contact them?• Are you talking too loudly?

Too quietly?• How much background

noise is there?

2

Photo source: whatisacellphonejammer.com

Mobile Phone Principles of Good IOR Behavior

3Strategic Use• Do you have the right

connections?• Are there new apps?• Are you playing too much Angry

Birds?• Rate/assess your experience

Mobile Phone Principles of Good IOR Behavior

Aware of Surroundings• How should others contact

you? You contact them?• Are you talking too loudly?

Too quietly?• How much background

noise is there?

Getting the Right Plan

• What do you need to accomplish?

• Do you have appropriate bandwidth (value vs. congestion)?

• Right amount of minutes/airtime?

Strategic Use• Do you have the right

connections?• Are there new apps?• Are you playing too much Angry

Birds?• Rate/assess your experience

21

3