Valuing Voices: Sustainable Solutions for Excellent Impact Jindra Cekan, PhD Valuing Voices at CEKAN CONSULTING LLC and Siobhan Green of Sonjara (partner.

Valuing Voices: Sustainable Solutions for Excellent Impact

Jindra Cekan, PhDValuing Voices at CEKAN CONSULTING LLC and

Siobhan Green of Sonjara (partner org)AEA, Denver – Oct 2014

Why do we fund – and evaluate- international development projects?

© 2014 Valuing Voices, Cekan Consulting LLC

We may want to: 1) Fulfill a mandate:

• Make the world a better place by providing much-needed funding and expertise to the needy

• Prove we spent taxpayer funds well2) Prove we did a great job…

• we were successful in using our inputs and outputs for sustained outcomes and impacts – or not

3) … or learn from where we haven’t?

4) Why else? ___________________How could we make aid more sustainable?

Why We Value Voices


Designing for sustained exit should be our first priority

We ALL deserve self-sustaining projects: Communities/ countries, we development workers, taxpayers… it’s true respect.

‘They’ should evaluate our assistance for effectiveness, not the other way around

The business model works – do more of what’s excellent/ profitable and less of what’s not. Currently in international development we don’t know much about sustained impact. Let’s change that and fund future projects accordingly

‘They’ are ‘Us’

What’s Great?


• Huge funding! US$20.4 billion 2014 USG funding and EU’s $1.5 trillion budget (2014-20)• Over 300 international non-profits in the US alone… helping 150+ countries over the last 50+ years!• Funding to local organizations by USAID is nearly 20%

Great M & E work is being done…


•Our industry is helping millions of ‘beneficiaries’ every year through multi-sectoral, global projects

• We desire real impact: USAID and EU have invested millions in M&E as have Gates, Rockefeller… but we’re evaluating impact- huh?

•IOCE supports over 100 national VOPEs (national evaluator associations) in 93 countries

•Donors have invested further millions in learning ventures such as 3ie (impact) and Challenges such as USAID’s/ Gates Foundation Grand (Health) Challenges, Making All Voices Count etc …

Global Commitment to Aid Effectiveness


US Government signed onto these goals in 2011:Paris Declaration (2005), Accra Agenda for Action (2008) and the Busan Partnership (Nov 2011) created shared principles to achieve common goals:

• Ownership of development priorities by developing counties: Countries should define the development model that they want to implement.• A focus on results: Having a sustainable impact should be the driving force behind investments and efforts in development policy making• Partnerships for development: Development depends on the participation of all actors, and recognizes the diversity and complementarity of their functions.•Transparency and shared responsibility: Development co-operation must be transparent and accountable to all citizens

http://www.oecd.org/dac/effectiveness/busanpartnership.htm

Development is becoming ripe for change…• Local Accountability has been proposed as a core feature of the new

post-2015 development agenda, according to UNICEF.• MDGs are becoming Sustainable Development Goals post-2015• Donors talk about Country-led Development, M&E Capacity (OECD),

Local Systems (USAID), as do non-profits like Local First- wonderful. Let’s support community-owned and driven ones!


http://post2015.org/tag/unicef/

http://sustainabledevelopment.un.org/?menu=1300

http://www.mymande.org/content/country-led-development-evaluation-donor-role-supporting-partner

http://valuingvoices.com/sustainability-springing-out-all-over-the-place-and-disrupting/

http://actlocalfirst.org/

Locating Valuing Voices in the sustainability evaluation landscape


Baseline, midterm and End of project evaluation

Impact evaluation (projected), Outcome Harvesting

Ex-post evaluation (actual)

Self-sustainability evaluation…

EVALUATION- Done with/ to

ICT4D Knowledge Retention/ Learning

Databases, e.g. DEC, 3ie, VOPE national databases

Done by

Paper documents Self-

generation Open-source database…

Done by

Mobiles/ tablets for crowd-sourcing

PVO does it all………………+Country/NGO does it………………..+Community does it all

Systems: Empowerment Eval, IOCE’s VOPEs

What could be improved?


“Real” Sustainable Development?


Standard Operating Procedurepost close out:• Billions of $$ invested• Ex-post evaluations not

performed, only few w/projected impact

• Work ‘transferred’ to weak NGOswithout any continuing funds

• Data + tracking systems discarded• National evaluations nonexistent • Minimal knowledge of projects by

national governmentsWe don’t know what worked after we leave because we don’t ask

Scant design for self-sustainability

Communities’ voices are also not elicited to design or to improveactivities they will make last…To not evaluate sustainability insults the communities which have entrusted us to improve their lives


Views from outside on our DIME*


"You cannot set your own examination, take the examination and mark it. Then you cry out success or failure. The community will just 'look at you' and wonder what is the issue.... It will be sustainable development if the people at community level are involved in designing and delivering their own dreams of development”

Peter Kimeu, Catholic Relief Services’ Senior Technical Advisor for Partnership/ Solidarity, East Africa (30 years expertise)

* Design, Implementation, M&E

Views from outside on our DIME*


• “We push new ideas, rather than building on what communities have already created…. we must be able to look back and see what they've learned and can sustain themselves.… to show what they've done themselves, and where they think they're taking themselves. [Sustainability] is an ignored area. Challenge is, it's just been gathering numbers for 5 years, next 3 years, 10 years.

• The implementer doing everything, leaves no space for communities…. It doesn't work because we don't have the right feedback system from communities to the NGO world. It’s the tyranny of deadlines so they "just get their concerns" when go out [to the field]… budgets and timing don't accommodate communities' ability to have time gathering together for distilled learning.”R.K. Kenyan Independent Evaluation Professional (20 yrs)

* Design, Implementation, M&E

What Exists Now …• There is a gaping hole in development and accountability: no long-term post-

project evaluation of expected impacts and what has emerged after projects end, much less community-determined definitions of desired impact;

• Sustainability is now narrowly defined as continued orgazinational financing; • Projects come and go, barely consult and withdraw resources in fixed times• Hardly ever are unintended impacts documented and learned from… + or -• Learning is often limited to within the projects, rarely across projects or later• Data and systems end at close-out, none open-source (rarely searchable)

Development Project, 2-5 year duration, typically

…The continuity of people’s lives in communities…

? ? ? ? ?Earlier or concurrentDevelopmentprojects

? ? ? ? ?

? ? ? ? ?

STOP?

baseline midterm finalE V A L U A T I O N S

Feedback loops?!


Feedback loops?


Rarely is there a problem for which there is such a clear solution:

• Shockingly, 99% of development projects are not evaluated for sustainability after close-out. (We’ve researched AEA, World Bank, USAID, EU databases…)

• Ask Communities what has worked best in international development, what they want more of, and can sustain themselves

• Do that everywhere. • That’s SUSTAINABILITY• Tell Others, Learn, Repeat.

What are our real clients’ ROI?• We need to explore the long-term Return on Investment

(ROI) of projects to communities• Focusing on ROI to just donors’ taxpayers misses the

goal of development which is to Exit sustainably• Ignoring community constituent voice is:

– a symptom of a system focused on short-term success– a missed opportunity to learn from what was most self-

sustaining (and why!) – A doorway to improve ways to co-design for self- sustainability as that is the real point in most development projects (with theexception of the needs for short-term humanitarian aid)© 2014 Valuing Voices, Cekan Consulting LLC

Why is this Imperative Now?


• US$20.4 billion 2014 USG funding and EU’s $1trillion budget is unevaluated for sustainability (www.ValuingVoices.com/blogs)

• Learning from post-project will decrease waste, increase funding and increase quality of programming and transparency/ accountability.

• Post-project evaluation will give organizations greater legitimacy in the eyes of both donors and communities and could secure more funding in the future

• Such evaluation of our ROI (return on investment) helps us finish ‘development’ by making ourselves obsolete, to build national capacity, and help them sustain the best, long after we do leave

• Main financial issue that blocks ex-post is that donor funding ends with the project. So set a trust fund of 5% M&E or a funded initiative such as 3ie (impact evaluations)

http://www.ValuingVoices.com/blogs

Current barriers:


• Lack of donor focus to evaluate long-term aid effectiveness; low funding due to organizational/ bureaucratic inflexibility to fund projects beyond project close-out

• Narrow vision that tracking success ends with project log frame parameters… rather than long-term opinions of our ‘targets’

• Too narrow use of data and technology without conserving, only extracting + exit

Feedback: how right are we?


• Are you too, seeing the focus being limited to accountability to donors rather than to communities for long-term impact?• Sustainability:

• What’s so Right you have to celebrate? • What’s so Wrong you want to scream?

• Where is data capture, project learning and knowledge retention and sharing being done Well?• Where is it Worse than we think?

And while…


…

Leadership needed: ours! Unless we Value Participants’ Voices… to evaluate projects, make them their own:


• Will we continue with narrowly focused, relatively-solvable projects aimed at the symptoms of underdevelopment as we have now? •… or look to learn from (un) expected impacts 2, 3, 5 after projects end? Ask participants what they feel projects yielded?

What if we:

What Can Be Done?


Accountability to our True Clients


This is what communities will measure themselves in Ethiopian Red Cross’ Tigray food security project:


Self-Sustainability Impact

VALUING VOICES evaluated self-sustainability of ERCS food security project through Community Participatory M&E

CURRENTLY NO ONE SYSTEMATICALLY, REPEATEDLY RETURNS TO EVALUATE POST-PROJECT SUSTAINABILITY…. But communities know what works

Valuing Voices’ Great Catalysts


Organization Project(s)

US Mercy Corps MILK Program in Niger; Community Recovery in Central Asia

US Plan International Kenyan Livelihood Project

Argentina's GVC OLNUS' Support to generation and consolidation of cooperatives of Argentine Puna

Japan's JICA Agricultural Development Project in Kambia District (Zambia)

PACT US/ Nepal Micro-credit (likely)

CARE US/ Food security (possible)

WHO ELSE?!


Examples of sustainability: ICT4DEvery project seems to have an app or website these days, to deliver services or capture information. Consider: • Technology: use mobiles or tablets for

participation or crowd-sourcing but take care to gauge access (gender, wealth, age, cost, literacy)

• Software sustainability- after funding ends, who maintains, updates, fixes and upgrades the app?

• Financial sustainability – who pays for the above? For new features and security fixes?

• Data sustainability – what happens to the data that is captured in the system after funding ends? Is the data available to other users (such as local stakeholders) for their usage? Is that data still protected?


What it looks like example: ICT4D• Partner with local ICT firms to design app with local

sustainable financing /business model– Local ICT firm focused on local users – responsive to local needs– Business model designed to cover both maintenance but also

future upgrades without donor funding– Pay per service can build usability expectations by making

beneficiaries also clients• Design data captured using open data international

standards (such as IATI)– Aggregate with other data sets (project, demographic)– Sharable beyond project staff/activities– Available after project period of performance– Allows for public accountability /transparency

Valuing Voices’ plan


1. Do primary and more secondary research

2. Pilot our approach (using a mix of empowerment evaluation, participatory appreciative inquiry, and community-led outcome harvesting in fieldwork in Uganda, Kenya, ___ (where else)?

3. Share results, test and replicate approach elsewhere, look for commonalities, celebrate diversity!

4. Advocate for funding and learningProof of concept is needed, using national evaluators asking communities, feeding into national databases, for global rollup, analysis and learning, feeding new design for more accountable, equitable and sustainable partnerships

More champions are needed!

Time is ripe to Learn via Valuing Voices (2014-15)


1. Research: Confirm the absence of ex-post, prep for ICT:Find/ learn from more international non-profits’, bilateral and

multilateral post-project evaluations: • Literature review of post-project evaluations

• Further interviews with evaluators and project leads on successful and unsuccessful post-project evaluation experiences with ‘constituent voice’ including methodology pros/cons

• Capture more “lessons learned” and pre-populate the digital repository of past final evaluations, geo-coded plus qualitative data input/ analysis

Time is ripe…. For Valuing Voices (2014-15)


2. Fieldwork Testing of methods:• Pilot a ‘franchise model’ of national evaluators (Expand on

IOCE’s national evaluator network + Universities)+ local ICTs and populate digital repository for analysis

• Identify best ways to build feedback cycles using traditional interviews and ICT methods plus local analysis w/nationals

• Pilot evaluation with WOUGNET (Women of Uganda’s Network) on closed and ongoing projects in ICT for Agriculture (300 interviewees: what activities they could sustain themselves, and why/ why not)Other pilots ready, e.g. Kenya

http://wougnet.org/


Time is ripe…. For Valuing Voices (2015)3. Share Learning

• Digital repository analysis (data set resource)• Presentations, blogs and trainings to Ugandan

government, our funders, US donors, evaluation and ICT experts, advocates• Franchise model e-rollout

4. Advocate for more funding, betterlocal and global learning • More can learn from local experts, rollup!• Focus on the data – to show impact• Advocate to donors to fund such workon more of the $1.5trillion of development projects now!


PILOT YEAR BUDGET: VALUING VOICES Research and Ugandan Fieldwork

1. Research and writing:

Staff time, travel: $140,000

2. Fieldwork• Staff (US, Ug, Ky): $60,000• Travel (US, Ky to Ug) $16,500• Consultations $33,500

TOTAL: $250,000…

Can the Development Industry

afford not to trial and adopt this

methodology, especially given missed

learning to date?

Any funders in the house ?

Feedback Request: Proposed ResearchSelection:• Ex-post project sites: projects closed within the last 2-3 years• Sector could be food security/ livelihoods, will select across age/gender • Past final evaluation/ financials on-hand, at minimum project design data• National databases in web-saavy countries to handle data/ analysis/ curation/

storage

Methodologies:• Possibly have a random-control site, e.g. one w/o further assistance vs. one with

further assistance, or comparable village without same aid• Feedback gathered from communities, by country national evaluators (pre-

trained)• Empowerment Evaluation principles, community-led• Outcome Harvesting looking for probable contribu-tion (not attribution) of inputs to impacts• Participatory/ Qualitative approaches (5-10 main questions of (un)expected impact• Appreciative Inquiry focus on what has worked best• ICT: mobile surveys for crowd-sourcing


Feedback (2): Learning best from what lasts?

@ 2014 Valuing Voices, Cekan Consulting LLC

Considerations/ processes•Whole of community discussions, 2 days per community•Feedback to local authorities and national Ministries•Feed post-project learning into new projects DIME in area•Q: How to best support learning from databases nationally?•Q: How best engage national decision-makers, donors inmacro feedback learning loops across projects, sectors?

Social: •Q: Will empowering communities to self-evaluate engage them and NGOs/PVOs in future project design?•Q: Will using national evaluators, feeding into national and global databases work for advocating fundamental change in how development is ‘done’ re: design and learning?•Q: How to attribute unintended impacts to vestige of project or not?•Q: how to address sharing sensitive learning of ‘failed’ projects? • How best to build the sensitive case If some sectors/ activities are far lessself-sustainable than others yet donors feel they must fund?• What else?

Barriers to this work of Valuing Voices?• Donor reluctance to question inflexible budget and funding cycles • Not enough national learning, knowledge retention • Difficulty devolving power/ authority to countries, focusing aid on demand, not supply• Option to listen to qualitative, local wisdom in addition to parallel post-project RCT impact evaluations• This is more than local NGO capacity building• Weakness in processes of some local government and

ministries to absorb lessons from such research, plus a dearth of available excellent national evaluators

• Slow internet connections and prohibitive costs of mobile ICT communications

• WHAT ELSE? What do we need for this to succeed?© 2014 Valuing Voices, Cekan Consulting LLC


Near-term projected Benefits of Valuing Voices?

Learning from post-closeout evaluations results in:

• Learning what enabled sustainability or prevented it… for other projects to do/ or not• A global database of post-project evaluation lessons by sector by

country, sector, even implementer on ROI by project investment from participants to donors/implementers to their taxpayers- how sustainably $$ was spent

• A cadre of trained national evaluators/ students and ICT database managers exists in each country

• Inspiring better local solutions via better information on what works best

• Fostering more sustainable, cheaper and resilient development through national Consumer Reports, AIDMaps etc.


What can we do now?1) Monitoring: Work with communities to co-create their own self-

monitoring system: IIED PLA Notes referenced in ValuingVoices blog, build this into baselines, midterm, final and beyond.

2) Ongoing evaluation via Feedback Loops: Support local evaluators to evaluate prospects for sustaining project impacts at design, during the midterm or at project’s end, give them feedback on how their input to the project was used, problem-solve together for impact.

3) Advocate to donors for community-led ex-post self-sustainability evaluation of actual impacts 2, 3, 5 years after projects close (community-felt intended and unintended) impacts, knowledgeretention and sharing via ICT, global databases, build capacity…

Don’t we want to be at the forefront of development?What’s missing to ‘DO DEVELOPMENT MORE SUSTAINABLY!?

http://valuingvoices.com/stepping-up-community-self-sustainability-one-ethiopian-step-at-a-time/

http://feedbacklabs.org/see-how-it-turned-out-feedback-loops-for-implementation-and-sustainability/

http://valuingvoices.com/pineapple-apple-what-differentiates-impact-from-self-sustainability-evaluations/

Let’s go!


Let’s change how international development works… together.

Thank you!

Valuing Voices: Sustainable Solutions for Excellent Impact Jindra Cekan, PhD Valuing Voices at CEKAN CONSULTING LLC and Siobhan Green of Sonjara (partner.

Documents

development model

development workers

development policy

phd valuing voices

development agenda mdgs

voices count

development co operation

sustainable impact