Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Making the most of it:Assessing impact with small data sets

Susanna Dilliplane, PhDDeputy Director

Aspen Planning and Evaluation Program

ANDE Annual Metrics ConferenceJune 23-24, 2015

Overview

What you can expect

What we’re going to do

What APEP does

Starting point:I want to assess my impact,

but I can’t run a large-scale RCT

When?

Random assignment isn’t possible

The intervention is changing over time

Impacts are not adequately captured by quantitative measures

Causal pathways are complex Small-scale projects We are not hypothesis testing

Alternatives?

Quantitative Surveys Documents/records

Qualitative Interviews Focus groups Site visits Open-ended survey questions Document review Participatory methods

Can we still say something meaningful about impact?

Can we still say something meaningful about impact?

Absolutely!

How do I optimize my ability to collect meaningful and

useful data on impact?

PrioritizeWho is your audience?

Decisions, decisions, decisions…

Am I looking at my impact on SGBs or my impact on the people the SGBs serve?

ANDE member does X Impact on SGB

Impact on individuals,

communities, etc.SGB does Y

Example:

Example:

Root Capital provides loans

to Freshco

Freshco is able to produce more high-

yield, drought-resistant maize

seed

Farmers have increased yield

(more food, increased income, surplus grain for local population)

Greater access to high-quality seed

for farmers

Am I looking at short-term or long-term impacts?

Am I in hypothesis testing mode or exploratory mode?

Did I have X impact? What impacts did I have?How or why did I have an

impact?

vs.

Make the most of your

measures

#1. Measure outcomes, not just outputs

Outputs Outcomes

Your products/services to SGBs. Impact.

#2. Not just “whether” but also “how” and “why”

Did I have an impact?Whether

How

Why

How did my activities contribute to a positive

impact?

Why didn’t I have an impact?

Vote with your

You’ve worked with quantitative

methods

You’ve worked with qualitative methods

#3. Consider qualitative and quantitative

Combine if possible!

QuantitativeQualitative

#4. Sample with purpose

#4. Sample with purpose

How big is big enough?

Sample size calculator:

http://enterprise-development.org/page/calculator

#5. Scout external factors

#6. Try for two (or more)

Time 1 Time 2

T1 T2 T3 T4 T5

#7. Optimize questionnaires

Is this clear? If everyone answers “yes,”

what will that tell me?

Ask good questions

Be mindful of burden

Beware of branching

Too many questions can lead to fatigue and skipping

Sub-groups may be too small to analyze

and share

Measuring outcomes rather than outputs

Measuring “how” or “why” Combining qualitative and

quantitative Sampling External influences Collecting data at multiple

points Questionnaires

Ok, I’ve collected my data.

Now what?

Data analysis:Things to check

Your sample

What’s my response rate? Anybody under/over-

represented? Does the pre match the

post? Are there missing data?

Analysis

Does it make sense to calculate statistical significance?

Should you disaggregate?

What’s the appropriate way to analyze your qualitative data?

Is that calculation

meaningful?

Assess yourlimits Attribution vs. Contribution

Counterfactuals

Generalizability

What to do when…

Group Exercise