Top Banner
Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual Metrics Conference June 23-24, 2015
33

Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Dec 28, 2015

Download

Documents

Randolf Merritt
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Making the most of it:Assessing impact with small data sets

Susanna Dilliplane, PhDDeputy Director

Aspen Planning and Evaluation Program

ANDE Annual Metrics ConferenceJune 23-24, 2015

Page 2: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Overview

What you can expect

What we’re going to do

What APEP does

Page 3: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Starting point:I want to assess my impact,

but I can’t run a large-scale RCT

Page 4: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

When?

Random assignment isn’t possible

The intervention is changing over time

Impacts are not adequately captured by quantitative measures

Causal pathways are complex Small-scale projects We are not hypothesis testing

Page 5: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Alternatives?

Quantitative Surveys Documents/records

Qualitative Interviews Focus groups Site visits Open-ended survey questions Document review Participatory methods

Page 6: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Can we still say something meaningful about impact?

Page 7: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Can we still say something meaningful about impact?

Absolutely!

Page 8: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

How do I optimize my ability to collect meaningful and

useful data on impact?

Page 9: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

PrioritizeWho is your audience?

Page 10: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Decisions, decisions, decisions…

Page 11: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Am I looking at my impact on SGBs or my impact on the people the SGBs serve?

ANDE member does X Impact on SGB

Impact on individuals,

communities, etc.SGB does Y

Page 12: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Example:

Page 13: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Example:

Root Capital provides loans

to Freshco

Freshco is able to produce more high-

yield, drought-resistant maize

seed

Farmers have increased yield

(more food, increased income, surplus grain for local population)

Greater access to high-quality seed

for farmers

Page 14: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Am I looking at short-term or long-term impacts?

Page 15: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Am I in hypothesis testing mode or exploratory mode?

Did I have X impact? What impacts did I have?How or why did I have an

impact?

vs.

Page 16: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Make the most of your

measures

Page 17: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

#1. Measure outcomes, not just outputs

Outputs Outcomes

Your products/services to SGBs. Impact.

Page 18: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

#2. Not just “whether” but also “how” and “why”

Did I have an impact?Whether

How

Why

How did my activities contribute to a positive

impact?

Why didn’t I have an impact?

Page 19: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Vote with your

You’ve worked with quantitative

methods

You’ve worked with qualitative methods

Page 20: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

#3. Consider qualitative and quantitative

Combine if possible!

QuantitativeQualitative

Page 21: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

#4. Sample with purpose

Page 22: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

#4. Sample with purpose

How big is big enough?

Sample size calculator:

http://enterprise-development.org/page/calculator

Page 23: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

#5. Scout external factors

Page 24: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

#6. Try for two (or more)

Time 1 Time 2

T1 T2 T3 T4 T5

Page 25: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

#7. Optimize questionnaires

Is this clear? If everyone answers “yes,”

what will that tell me?

Ask good questions

Be mindful of burden

Beware of branching

Too many questions can lead to fatigue and skipping

Sub-groups may be too small to analyze

Page 26: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

and share

Measuring outcomes rather than outputs

Measuring “how” or “why” Combining qualitative and

quantitative Sampling External influences Collecting data at multiple

points Questionnaires

Page 27: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Ok, I’ve collected my data.

Now what?

Page 28: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Data analysis:Things to check

Page 29: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Your sample

What’s my response rate? Anybody under/over-

represented? Does the pre match the

post? Are there missing data?

Page 30: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Analysis

Does it make sense to calculate statistical significance?

Should you disaggregate?

What’s the appropriate way to analyze your qualitative data?

Is that calculation

meaningful?

Page 31: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Assess yourlimits Attribution vs. Contribution

Counterfactuals

Generalizability

Page 32: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

What to do when…

Page 33: Making the most of it: Assessing impact with small data sets Susanna Dilliplane, PhD Deputy Director Aspen Planning and Evaluation Program ANDE Annual.

Group Exercise