Program theory and logic models
for systemic evaluation
International Conference on Systemic Approaches in Evaluation
Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ)
Eschborn, Germany 25-26 January 2011
Patricia Rogers [email protected]
2
Overview
• Presentation:
– Simple, complicated and complex systems
– 7 aspects to consider for evaluation
• Examples
– How these aspects have been addressed in some recent examples of program theory
• Discussion
What recommendations of practical relevance can be made?
– How to develop it further?
– How to create the preconditions for systemic evaluations (if it is considered desirable)
Recommendations for
– Evaluators
– Managers of evaluations
– Donors
3
Sources for this presentation
4Funnell and Rogers Purposeful Program Theory 4
Program theory An explicit theory of how an intervention contributes to the
intended or observed outcomes, which has 2 components:
Theory of
change
The process by which change comes about (for an
individual, organization or community)
Theory of action How the intervention is constructed to activate the theory of
change
Logic model A visual representation of a program theory, usually in a
diagram but sometimes in a table
Program theory
evaluation
An evaluation that is at least partly guided by an explicit
program theory. It is not necessarily „driven‟ by the theory,
since it should be driven by its intended purpose and the
needs of its intended users
Some definitions
5Funnell and Rogers Purposeful Program Theory 5
What people sometimes assume you mean by logic models that
address complexity
66
Two framings of simple, complicated and complex
Glouberman and Zimmerman 2002 Kurtz and Snowden 2003
Simple Tested „recipes‟ assure
replicability
Expertise is not needed
The domain of the „known‟,
Cause and effect are well
understood,
Best practices can be confidently
recommended,
Complicated Success requires high level of
expertise in many specialized
fields + coordination
The domain of the „knowable‟
Expert knowledge is required,
Complex Every situation is unique –
previous success does not
guarantee success
Expertise can help but is not
sufficient; relationships are key
The domain of the „unknowable‟,
Patterns are only evident in
retrospect.
Glouberman, S., and Zimmerman, B. Complicated and Complex Systems: What Would Successful Reform of Medicare Look Like?
Ottawa: Commission on the Future of Health Care in Canada, 2002. http://www.healthandeverything.org/fi les/Glouberman_E.pdf.
Kurtz, C. F. and D. J. Snowden (2003) „The New Dynamics of Strategy: Sense-making in a Complex and Complicated World‟, IBM
Systems Journal 42(3): 462–83. ( who also discuss chaotic and disordered)
7
Different types of systems
A simple system
A complicated system
A complex system
1. http://commons.wikimedia.org/wiki/File:Ford_assembly_line_-_1913.jpg 2 .Heath Robinson 3. Oriolus
http://www.flickr.com/photos/28556257@N00/441814998/
8
What constitutes a systemic approach to evaluation?
Attention to:
• Inter-relationships
• Perspectives
• Boundaries
Williams B. Imam I. (2007) (eds) Systems Concepts in Evaluation - An Expert Anthology
EdgePress/AEA Point Reyes CA.
Williams, B and Hummelbrunner, R (2010) Systems Concepts In Action: A Practitioner's Toolkit„
Stanford: Stanford University Press.
9
Funnell, S.C. and Rogers, P.J. (2011) Purposeful Program Theory. San Francisco: Jossey-Bass/Wiley.
9
Aspects of complicated and complex situations and interventions with
potentially important implications for evaluation
1) Focus
2) Governance
3) Consistency
4) Necessariness
5) Sufficiency
6) Change trajectory
7) Unintended outcomes
1010
(1) Focus
Simple Single set of intended outcomes/impacts
Complicated Different intended outcomes/impacts intended by
different partners/stakeholders
Different intended outcomes/impacts at different levels
Complex Emergent intended outcomes/impacts
1111
Simple focus
Intervention Longer term
outcomes
Shorter term
outcomes
Intervention that produces single set of outcomes
1212
Complicated focus (1)
Intervention that produces different outcomes valued by different
stakeholders
INTERVENTION SHORT-TERM
OUTCOMES
LONG-TERM OUTCOMES
DIFFERENT LONG-TERM OUTCOMES
1313
Intervention Longer term
outcomes Shorter term outcomes at
system level
Activities at
system level
Activities at site
level
Activities at
client level
Shorter term outcomes at
site level
Shorter term outcomes at
client level
Complicated focus (2)
Intervention that produces different outcomes at different levels
14Funnell and Rogers Purposeful Program Theory 14
Complex focus (1)
Intervention that produces emergent intermediate outcomes
15Funnell and Rogers Purposeful Program Theory 15
Complex focus (2)
Intervention that produces emergent long-term outcomes
16Funnell and Rogers Purposeful Program Theory 16
(2) Governance
Simple Single organization
Complicated Specific organizations with formalized requirements
Complex Emergent organizations working together in flexible
ways
17
(3) Consistency
Simple Implement what has been identified as „best practice‟ or
„evidence-based practice‟ – what works
Complicated Classify the situation and implement what has been
adapted for that context – what works for whom in what
situation
Complex Ongoing adaptation to emerging conditions – what is
working here
18
Complex consistency
Adaptive, responsive intervention
SHORT-TERM OUTCOMES
LONG-TERM OUTCOMES
INTERVENTION
19Funnell and Rogers Purposeful Program Theory 19
(4) Necessariness
Simple Only way to achieve the intended impacts
Complicated One of several ways to achieve the intended impacts – which can be
identified in advance
Complex One of several ways to achieve the intended impacts – which are only
evident in retrospect
20Funnell and Rogers Purposeful Program Theory 20
(5) Sufficiency
Simple Sufficient to produce the intended impacts. Works the same for
everyone
Complicated Only works in conjunction with other interventions (previously,
concurrently, or subsequently) and/or only works for some people
and/or only works in some circumstances – which can be identified
in advance
Complex Only works in conjunction with other interventions (previously,
concurrently, or subsequently) and/or only works for some people
and/or only works in some circumstances – which is only evident in
retrospect
Funnell and Rogers 2010 Purposeful Program Theory. Jossey-Bass)
21Funnell and Rogers Purposeful Program Theory 21
Ways in which an intervention can work with other interventions
Stronger Families and Communities Strategy evaluation 2000-2004 Final Reporthttp://www.rmit.edu.au/casr/sfcse
22Funnell and Rogers Purposeful Program Theory 22
Complicated necessariness (1)
Intervention
Longer term
outcomes
Shorter term
outcomes
A different
intervention
Multi-stage intervention (eg Outcome Mapping)
23Funnell and Rogers Purposeful Program Theory 23
Complicated necessariness (2)
INTERVENTION SHORT-TERM
OUTCOMES
OUTCOMES/ IMPACTS
ANOTHER INTERVENTION
SHORT-TERM OUTCOMES
ANOTHER INTERVENTION
SHORT-TERM OUTCOMES
Intervention that works in combination with other interventions
24Funnell and Rogers Purposeful Program Theory 24
Complicated necessariness (3)
INTERVENTION SHORT-TERM
OUTCOMES
PARTICULAR PARTICIPANT CHARACTERISTICS
LONG-TERM OUTCOMES
INTERVENTION
DIFFERENT PARTICIPANT CHARACTERISTICS
DIFFERENT SHORT-TERM OUTCOMES
DIFFERENT LONG-TERM OUTCOMES
Intervention that works differently for different types of participants
2525
(6) Change trajectory
Simple Constant, linear relationship between effort and results (eg
twice the investment produces twice the results)
Complicated Well understood but not linear relationship between effort and
results (eg curvilinear dose-response relationship such as
diminishing returns or too much of a good thing)
Complex Emergent relationship between effort and results (eg unknown
tipping points)
26Funnell and Rogers Purposeful Program Theory 26
(7) Unintended outcomes
Simple Unintended outcomes can be anticipated and monitored
Complicated Different unintended outcomes are likely in particular
combinations of circumstances – expertise is needed to
anticipate them and identify them
Complex Unintended outcomes cannot be anticipated but only
identified (and addressed) as they emerge or in retrospect
Funnell and Rogers 2010 Purposeful Program Theory. Jossey-Bass)
27Funnell and Rogers Purposeful Program Theory 27
Some thoughts on how program theory might address
complicated and complex aspects
Issues that may need to
be addressed
1. Focus
2. Governance
3. Consistency
4. Necessariness
5. Sufficiency
6. Change trajectory
7. Unintended
outcomes
Possible evaluation methods, approaches and
methodologies
• Emergent evaluation design that can
accommodate emergent program objectives and
emergent evaluation issues
• Collaborative evaluation across different
stakeholders and organisations
• Non-experimental approaches to causal
attribution/contribution that don‟t rely on a
standardized „treatment‟
• Realist evaluation that pays attention to the
contexts in which causal mechanisms operate
• Realist synthesis that can integrate diverse
evidence (including credible single case studies)
in different contexts
• „Butterfly nets‟ to catch unanticipated results
28
Examples
Examples are on a separate handout, along with pages for recording
comments in terms of the 7 aspects and broader implications
29
1. VECO Indonesia: Sustainable Agricultural Chain
Development, Deprez and Van Steenkiste (2010)
30
2. New Zealand Department of Labor: Recognised Seasonal
Employer policy (Nunns and Roorda, 2009)
31
3. International Center
for Tropical
Agriculture (CIAT)
Striga programs
(Douthwaite, Kuby,
van de Fliert and
Schulz, 2003)
32
4. Waawiyeya Evaluation Tool, Tending the Fire
program (Johnston, 2010)
33
5. Stronger Families and Communities Strategy (CIRCLE, 2006)
CIRCLE „Third Newsletter. Evaluation of the Stronger
Families and Communities Strategy 2000-2004.
Funnell, S.C., Rogers, P.J. and Scougall, J. „Issues Paper on
Community Capacity Building for the Evaluation of the
SFCS‟. Canberra: Department of Family and Community
Services.
Both available at http://rmit.edu.au/casr/sfcse
34
Discussions
What recommendations of practical relevance can be made?
– How to develop it further?
– How to create the preconditions for systemic evaluations (if it is considered
desirable)
Recommendations for
– Evaluators
– Managers of evaluations
– Donors
35
References and further readingFunnell, S. and Rogers, P.J. (2011) Purposeful Program Theory. San Francisco: Jossey-
Bass/Wiley.
Glouberman, S., and Zimmerman, B. Complicated and Complex Systems: What Would Successful Reform of Medicare Look Like? Ottawa: Commission on the Future of Health Care in Canada, 2002. http://www.healthandeverything.org/fi les/Glouberman_E.pdf.
Kurtz, C. F., and Snowden, D. F. “The New Dynamics of Strategy: Sense-Making in a Complex and Complicated World.” IBM Systems Journal, 2003, 42(3), 462–483.
Patton, M. 2010 Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: Guildford Press.
Rogers, P. J. “Using Programme Theory for Complicated and Complex Programmes.” Evaluation, 2008, 14 (1), 29–48.
Rogers, P. J., Guijt, I., and Williams, B. “Thinking Systemically: Seeing from Simple to Complex in Impact Evaluation.” Presented at the 3IE/African Evaluation Association Impact Evaluation Conference, Cairo, Egypt, 2009.
Rogers, P.J. and Williams, B. 2010 „‟Using Systems Concepts in Evaluation‟‟ in Beyond Logframe; Using Systems Concepts in Evaluation. Tokyo: FASID http://www.fasid.or.jp/shuppan/hokokusho/pdf/h21-3.pdf
Snowden, D. J., and Boone, M. “A Leader‟s Framework for Decision Making.” Harvard Business Review, Nov. 2007, pp. 69–76.
Williams B. Imam I. (2007) (eds) Systems Concepts in Evaluation - An Expert Anthology EdgePress/AEA Point Reyes CA.
Williams, B and Hummelbrunner, R (2010) Systems Concepts In Action: A Practitioner's Toolkit„ Stanford: Stanford University Press