Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual Metrics Conference June 23-24, 2015
Dec 28, 2015
Making the most of it:Assessing impact with small data sets
Susanna Dilliplane, PhDDeputy Director
Aspen Planning and Evaluation Program
ANDE Annual Metrics ConferenceJune 23-24, 2015
Overview
What you can expect
What we’re going to do
What APEP does
Starting point:I want to assess my impact,
but I can’t run a large-scale RCT
When?
Random assignment isn’t possible
The intervention is changing over time
Impacts are not adequately captured by quantitative measures
Causal pathways are complex Small-scale projects We are not hypothesis testing
Alternatives?
Quantitative Surveys Documents/records
Qualitative Interviews Focus groups Site visits Open-ended survey questions Document review Participatory methods
Can we still say something meaningful about impact?
Can we still say something meaningful about impact?
Absolutely!
How do I optimize my ability to collect meaningful and
useful data on impact?
PrioritizeWho is your audience?
Decisions, decisions, decisions…
Am I looking at my impact on SGBs or my impact on the people the SGBs serve?
ANDE member does X Impact on SGB
Impact on individuals,
communities, etc.SGB does Y
Example:
Example:
Root Capital provides loans
to Freshco
Freshco is able to produce more high-
yield, drought-resistant maize
seed
Farmers have increased yield
(more food, increased income, surplus grain for local population)
Greater access to high-quality seed
for farmers
Am I looking at short-term or long-term impacts?
Am I in hypothesis testing mode or exploratory mode?
Did I have X impact? What impacts did I have?How or why did I have an
impact?
vs.
Make the most of your
measures
#1. Measure outcomes, not just outputs
Outputs Outcomes
Your products/services to SGBs. Impact.
#2. Not just “whether” but also “how” and “why”
Did I have an impact?Whether
How
Why
How did my activities contribute to a positive
impact?
Why didn’t I have an impact?
Vote with your
You’ve worked with quantitative
methods
You’ve worked with qualitative methods
#3. Consider qualitative and quantitative
Combine if possible!
QuantitativeQualitative
#4. Sample with purpose
#4. Sample with purpose
How big is big enough?
Sample size calculator:
http://enterprise-development.org/page/calculator
#5. Scout external factors
#6. Try for two (or more)
Time 1 Time 2
T1 T2 T3 T4 T5
#7. Optimize questionnaires
Is this clear? If everyone answers “yes,”
what will that tell me?
Ask good questions
Be mindful of burden
Beware of branching
Too many questions can lead to fatigue and skipping
Sub-groups may be too small to analyze
and share
Measuring outcomes rather than outputs
Measuring “how” or “why” Combining qualitative and
quantitative Sampling External influences Collecting data at multiple
points Questionnaires
Ok, I’ve collected my data.
Now what?
Data analysis:Things to check
Your sample
What’s my response rate? Anybody under/over-
represented? Does the pre match the
post? Are there missing data?
Analysis
Does it make sense to calculate statistical significance?
Should you disaggregate?
What’s the appropriate way to analyze your qualitative data?
Is that calculation
meaningful?
Assess yourlimits Attribution vs. Contribution
Counterfactuals
Generalizability
What to do when…
Group Exercise