Thinking systemically: Thinking systemically: Seeing from simple to Seeing from simple to complex in complex in impact impact evaluation evaluation Professor Patricia Rogers Royal Melbourne Institute of Technology, Australia Dr. Irene Guijt Learning by Design, the Netherlands Bob Williams Independent consultant, New Zealand (with thanks to Dr. Jim Woodhill, Wageningen International, the Netherlands) Expert lecture for AfREA Conference Expert lecture for AfREA Conference March 30 – April 2, 2009 March 30 – April 2, 2009 Cairo, Egypt Cairo, Egypt
31
Embed
Thinking systemically: Seeing from simple to complex in impact evaluation Professor Patricia Rogers Royal Melbourne Institute of Technology, Australia.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Thinking systemically: Thinking systemically: Seeing from simple to complex Seeing from simple to complex
in in impact evaluationimpact evaluation
Professor Patricia RogersRoyal Melbourne Institute of Technology, Australia
Dr. Irene GuijtLearning by Design, the Netherlands
Bob WilliamsIndependent consultant, New Zealand
(with thanks to Dr. Jim Woodhill, Wageningen International, the Netherlands)
Expert lecture for AfREA Conference Expert lecture for AfREA Conference March 30 – April 2, 2009March 30 – April 2, 2009 Cairo, EgyptCairo, Egypt
TODAY’S SESSIONTODAY’S SESSION
• Explore what thinking systemically is and how it relates to evaluation
• Introduce a systems approach that we think has the potential to move IDE forward, plus give you something you can use in your own practice
• Give you an opportunity to reflect and play with systems ideas and this method.
SYSTEMS CONCEPTS IN EVALUATIONAN EXPERT ANTHOLOGY
Eds. Iraj Imam & Bob Williams
http://www.iigss.net/gPICT.pdf
INTER-RELATIONSHIPS
PERSPECTIVES
BOUNDARIES
THREE ELEMENTS OF THINKING SYSTEMICALLY
INTER-RELATIONSHIPS
Being deeply aware of their significance
Some inter-relationships matter more than most
Some only matter over time
Some are slower in their impact than others
Some are linear (A affects B), some are non-linear and recursive (A affects B which affects A)
Most critically, systems thinking focuses on the inter-relationship of ideas, assumptions, beliefs as well as actions in the traditional cause and effect.
PERSPECTIVES
Thinking systemically about perspectives is not the same as stakeholder analysis
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
PERSPECTIVES
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
PERSPECTIVES We all bring different perspectives to
bear on anything we do. In this workshop I am handling four different perspectives
1. A session where people learn something
2. Something that allows me to communicate my knowledge
3. A means of expressing friendship and support to colleagues,
4. A way of enjoying myself.
You cannot understand how I behave at this session unless you understand how I juggle these perspectives.
QuickTime™ and a decompressor
are needed to see this picture.
BOUNDARIES
BOUNDARIES
Who or what is “in” and who or what is “out”
1. Purpose of evaluation; how will you judge “success”?
2. Resources for evaluation; what is not in your control?
3. What evidence is considered credible, whose expertise is acknowledged, or ignored?
4. Whose or what interests are not being served by an evaluation ?
BOUNDARIES
Thinking systemically requires you to do two things around those boundary decisions;
1.to identify the consequences of boundary decisions
2.consider how to mitigate any negative consequences of boundary decisions.
Tool for thought - the Cynefin FrameworkTool for thought - the Cynefin Framework
Facilitates seeing situational diversity Based on recognizing different types of cause
and effect relations – a given situation will contain aspects of all
Draws on theories of: Complexity Cognitive Systems Narrative Networks
Developed by Dave Snowden – ex-IBM knowledge management researcher
The Cynefin Framework – knowing The Cynefin Framework – knowing what you are dealing withwhat you are dealing with
‘‘Life is a path you beat by walking’ Life is a path you beat by walking’ complex impactscomplex impacts
Intermediate Results (at 1, t+1, t+2)
Plan A
Plan C Plan D Plan D2 Plan E Plan E2
Plan D2
Plan F
ConceptualisingConceptualising the Intervention the Intervention
‘Silver bullet’ impactsintervention is necessary and sufficient to produce impact
Counterfactual approaches comparing similar groups with/without the intervention
‘Ducks lined up’ impactsintervention only produces impact with other things in place
Need to compare with and without full causal package
‘Multiple path’ impactsimpacts achievable by different means
Need to compare alternative ways of achieving impacts
‘Complex’ impacts –achieved by right combination for that particular situation
Need to assess quality and impact of non-standardised interventions
Differentiated study of causality
So what is ‘impact evaluation’?
SIMPLE COMPLICATED COMPLEX
Question answered
What works? What works for whom in what contexts?
What’s working?
Process needed Knowledge transfer
Knowledge translation Knowledge generation
Nature of direction
Single way to do it
Contingent Dynamic and emergent
Metaphor for direction
Written directions
Map and timetable Compass
Take home messagesTake home messages
1. Seeing ‘ontological diversity’ in situations (and interventions) enables a more conscious, appropriate methodologically mixed approach – it’s about being a good professional.
2. Cynefin is just a heuristic - a tool for thinking systemically
3. Thinking systemically is about a deep understanding inter-relationships, perspectives and boundaries. Boundary critique is the area where evaluation can learn most by drawing on the experience of the systems field.
ReferencesReferencesEoyang, Glenda. (2008) So, what about accountability?
http://www.cognitive-edge.com/blogs/guest/2008/12/so_what_about_accountability_1.php Glouberman, S. and Zimmerman, B. (2002) Complicated and Complex Systems:What Would Successful Reform of
Medicare Look Like? Commission on the Future of Health Care in Canada. Discussion Paper 8. Available at http://www.healthandeverything.org/pubs/Glouberman_E.pdf
Guijt, I. (2008). Navigating Complexity. Report of an Innovation Dialogue, May 2008. http://portals.wi.wur.nl/files/docs/Innovation%20Dialogue%20on%20Navigating%20Complexity%20-%20Full
%20Report.pdf
Guijt, I. and P. Engel. (2009). ‘Nine Hot Potatoes. Current Debates and Issues in Results-Oriented Practice.’ Presentation for Hivos In-house Training on Reflection-oriented Practice.
Mackie, J. (1974). The Cement of the Universe. Oxford University Press, Oxford.Mark MR. 2001. What works and how can we tell? Evaluation Seminar 2. Victoria Department of Natural Resources
and Environment.Rogers, P.J. (2008) ‘Using programme theory for complicated and complex programmes’ Evaluation: the international
jourmal of theory, research and practice. 14 (1): 29-48.Rogers, P.J. (2008) ‘Impact Evaluation Guidance. Subgroup 2’. Meeting of NONIE (Network of Networks on Impact
Evaluation), Washington, DC.Rogers, P.J. (2001) Impact Evaluation Research Report Department of Natural Resources and Environment, Victoria. Ross, H. L., Campbell, D. T., & Glass, G. V (1970). Determining the social effects of a legal reform. In S. S. Nagel
(Ed.), Law and social change (pp. 15-32). Beverly Hills, CA: SAGE.Williams, B. and I. Imam. (2007). Systems Concepts in Evaluation: An Expert Anthology. American Evaluation
Association.Woodhill, J. (2008). The Cynefin Framework: What to do about complexity? Implications for Learning, Participation,
Strategy and Leadership. Presentation for the ‘Navigating Complexity Workshop’, Wageningen International. http://portals.wi.wur.nl/files/docs/File/navigatingcomplexity/CynefinFramework%20final%20%5BRead-Only%5D.pdf