Evaluation’s influence - the dream, the nightmare, and some waking thoughts Australasian Evaluation Society International Conference Keynote Address Sydney,

Evaluations influence - the dream, the nightmare, and some waking thoughts Australasian Evaluation Society International Conference Keynote Address Sydney, Australia 1 September 2011 Patricia Rogers 2 Discussions about evaluation and complexity IBM Systems Journal, The new dynamics of strategy: Sense-making in a complex and complicated world Cynthia Kurtz Dave Snowden Commission of the Future of Health Care in Canada Discussion Paper No. 8 Complicated and Complex Systems What would successful reform of Medicare look like? Sholom Glouberman Brenda Zimmerman 2007 Harvard Business Review Leader's Framework for Decision Making Dave Snowden Mary Boone 2008 Evaluation in Complex Adaptive Systems Glenda Eoyang Thomas Berkas 2007 Getting to Maybe Frances Westley Brenda Zimmerman Michael Quinn Patton Realistic Evaluation Ray Pawson Nick Tilley Evidence-Based Policy A Realist Perspective Ray Pawson Evaluation Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions Patricia Rogers 2008 3 Sources for this thinking about evaluation and complexity 2010 Developmental evaluation Michael Quinn Patton 2011 Purposeful Program Theory Sue Funnell Patricia Rogers NORAD conference Evaluating the complex Oslo, Norway Evaluating the complex Kim Forss, MIta Marra, Robert Schwarz (eds) 2008 Exploring the science of complexity: Ideas and implications for development and humanitarian efforts. ODI Working Paper 285 Ben Ramalingan Harry Jones 2008 Workshop, Cali, Columbia 44 Two framings of simple, complicated and complex Glouberman and Zimmerman 2002Kurtz and Snowden 2003 SimpleTested recipes assure replicability Expertise is not needed The domain of the known, Cause and effect are well understood, Best practices can be confidently recommended, ComplicatedSuccess requires high level of expertise in many specialized fields + coordination The domain of the knowable Expert knowledge is required, ComplexEvery situation is unique previous success does not guarantee success Expertise can help but is not sufficient; relationships are key The domain of the unknowable, Patterns are only evident in retrospect. Glouberman, S., and Zimmerman, B. Complicated and Complex Systems: What Would Successful Reform of Medicare Look Like? Ottawa: Commission on the Future of Health Care in Canada, les/Glouberman_E.pdf.http://www.healthandeverything.org/fi les/Glouberman_E.pdf Kurtz, C. F. and D. J. Snowden (2003) The New Dynamics of Strategy: Sense-making in a Complex and Complicated World, IBM Systems Journal 42(3): 46283. ( who also discuss chaotic and disordered) 5 Evaluation influence for simple aspects of interventions What interventions look likeDiscrete, standardized intervention How interventions workPretty much the same everywhere Questions asked in evaluationWhat works? Are we doing it right? Nature of advice given by evaluationSingle way to do it Best practices Process needed for evaluation influence Knowledge transfer Metaphor for evaluation influenceGoogle directions (one way to do it little skill needed to follow instructions) 6 Impact evaluation for complicated aspects of interventions What interventions look likeDifferent in different situations How interventions workDifferently in different situations (different people or different implementation environments) Questions asked in evaluationWhat works for whom in what contexts? Nature of advice given by evaluationContingent Good practices in particular situations Process needed for evaluation influence Knowledge translation to new situations Metaphor for evaluation influenceTransport map and timetable (need some skill to choose the most appropriate option for that time and place) 7 Impact evaluation for complex aspects of interventions What interventions look likeNon standardized and changing, adaptive, and emergent How interventions workResults sensitive to initial conditions as well as to context, generalisations rapidly decay Questions asked in evaluationWhat is working? Nature of advice given by evaluationDynamic and emergent Principles Process needed for evaluation influence Ongoing knowledge generation Metaphor for evaluation influenceTopographical map and compass (need to work it out as you go along) 8 9 Positive influence No influence Negative influence The dreamThe nightmares 10 Positive influence 11 Good evaluation provides.. Which can lead to Accurate, timely and credible information that identifies and explains poor performance Fixing problems Accurate, timely and credible information that identifies and explains good performance Reinforcing, supporting, repeating, replicating, expanding good practice Clear signals about what is important More focus (attention and resources) on priority issues Clear accountability and consequences Increases motivation to find ways to improve performance Increased ability to generate and use information Ongoing capacity to learn 12 4 common barriers to positive evaluation influence 1.Technical limitations of available evidence, delays in feedback, uncertainty/disagreement about what is needed in the intervention and the evaluation OUTCOMES EVALUATION (INCLUDING ALL FORMS OF EVALUATION FROM NEEDS ASSESSMENT, PROGRAM DESIGN, PROCESS EVALUATION, MONITORING AND IMPACT ASSESSMENT) OUTPUTSACTIVITIESINPUTS DELAY 13 1.Technical limitations of available evidence, delays in feedback, uncertainty/disagreement about what is needed in the intervention and the evaluation 2.Cognitive taking in new information, overcoming assumptions "It is impossible for someone to learn what they think they already know." Epictetus (AD 55?-135?), Greek Stoic philosopher 4 common barriers to positive evaluation influence 14 4 common barriers to positive evaluation influence 1.Technical limitations of available evidence, delays in feedback, uncertainty/disagreement about what is needed in the intervention and the evaluation 2.Cognitive taking in new information, overcoming assumptions 3.Emotional defensive routines in response to shame, fear, and grief Great nations are like great people: when they make a mistake, they realize it; having realized it, they admit it; having admitted it, they correct it; they consider those who point out their faults as their most benevolent teachers. Lao Tzu 15 4 common barriers to positive evaluation influence 1.Technical limitations of available evidence, delays in feedback, uncertainty/disagreement about what is needed in the intervention and the evaluation 2.Cognitive taking in new information, overcoming assumptions 3.Emotional defensive routines in response to shame, fear, and grief 4.Organisational- incentives that support or restrict generation and use of information for improvement including organised self- interest, dysfunctional accountability systems and adversarial politics 16 No influence Replacing the $1 Note with a $1 Coin Would Provide a Financial Benefit to the Government March 2011 A Dollar Coin Could Save Millions. July Dollar Coin: Reintroduction Could Save Millions If It Replaced the 1-Dollar Note. May One-Dollar Coin: Reintroduction Could Save Millions if Properly Managed. March A New Dollar Coin Has Budgetary Savings Potential but Questionable Acceptability. Washington, D.C.: June According to GAOs analysis, replacing the $1 note with a $1 coin could save the government approximately $5.5 billion over 30 years. No influence 17 No influence 18 Instrumental use: Misleading leads to incorrect decisions about changes/continuation/termination due to: Error Data corruption Gaming Over-generalization (emphasis only on the average effect) Failure to appropriately address contributing factors Belief in the randomization fairy Ways that evaluation can and does have a negative influence Negative influence 19 Data corruption 2001 The company behind the Ambulance Emergency Dispatch Service illegally made phantom calls to boost its performance for financial gain Prosecutors to review widespread cheating in Atlanta schools Cheat and exclusion claims rock literacy tests 2008 Victorian hospitals manipulating data, admitting patients to "virtual wards", and inconsistently measuring waiting times to meet Government benchmarks for bonus payments. 20 Data corruption Strategies used to manipulate results in drug trials: choice of placebo as comparator selection of subjects (Bodenheimer, 2000) manipulation of doses (Angell, 2004). method of drug administration (Bodenheimer, 2000). manipulation of timescales ( Pollack & Abelson, 2006). suspect statistical analysis deceptive publication suppression of negative results ( Mathews, 2005) selective publishing (Mathews, 2005, Armstrong, 2006; Harris, 2006; Mathews, 2005; Zimmerman & Tomsho, 2005) opportunistic data analysis (Bodenheimer, 2000) control of authorship (Bodenheimer, 2000) House, E. 2008, Blowback: the consequences of evaluation, American Journal of Evaluation, vol. 29, no. 4, 41626. ) 21 Evaluation influence understood as knowledge transfer between researchers, policymakers and practitioners RESEARCHERS POLICYMAKERS PRACTITIONERS FIND THAT THING A WORKS DECIDE TO DO THING A DO THING A SINGLE STUDY SEVERAL STUDIES 22 Evaluation influence understood as knowledge translation between researchers, policymakers and practitioners RESEARCHERS POLICYMAKERS PRACTITIONERS FIND THAT THING A WORKS DECIDE TO DO THING A DO THING A SINGLE STUDY SEVERAL STUDIES STUDIES MIGHT BE TOO NARROW AND IGNORE IMPORTANT EVIDENCE MIGHT NOT BE FEASIBLE IN OTHER LOCATIONS MIGHT NOT BE SCALEABLE TO MULTIPLE LOCATIONS THERE MIGHT BE DIFFERENTIAL EFFECTS THING A MIGHT ONLY WORK IN SOME CONTEXTS THERE MIGHT BE UNINTENDED NEGATIVE EFFECTS 23 A review of early intervention programs for disadvantaged children, found that that some evidence-based programs which were effective, on average, were not only ineffective but actually damaging for the most disadvantaged, even when properly implemented, and even when the level of participation in the service was analysed (Westhorp (2008) For example, the Early Head Start program was found to have unfavourable outcomes for children in families with high levels of demographic risk factors (Mathematica Policy Research Inc, 2002). Westhorp, G (2008) Development of Realist Evaluation Methods for Small Scale Community Based Settings Unpublished PhD Thesis, Nottingham Trent University Mathematica Policy Research Inc (2002). Making a Difference in the Lives of Infants and Toddlers and Their Families: The Impacts of Early Head Start, Vol 1. US Department of Health and Human Services. Risks of over-generalizing What Works example - Early Head Start 24 Belief in the randomization fairy Example: Random assignment makes the two groups statistically equivalent in all aspects other than access to treatment, with the result that only the difference in treatment can cause a difference in outcomes between them. (Smith & Sweetman, 2008) 25 How the randomization fairy can be a dangerous myth A study that compared results from RCTs to those from observational studies found that, while the overall average effect size was similar, results from RCTs were more varied. Some of the RCTs produced paradoxical findings (that is, in some of trials the interventions produced negative effects and in other trials the same intervention had positive effects) which could be explained by random variation between treatment and control groups in terms of contributing factors (Concato et al, 2000). Concato J., Shah M.P.H. and Horwitz R.I. (2000) "Randomized,Controlled Trials, Observational Studies, and the Hierarchy of ResearchDesigns", New England Journal of Medicine, 342, 25,pp.18871892, Worrall, J. (2002). What Evidence in Evidence-Based Medicine? Causality: Metaphysics and Methods: Technical Report 01/03. London: Centre for Philosophy of Natural and Social; Sciences, London School of Economics. If 200 potted plants are randomly assigned to either a treatment group that receives daily water, or to a control that receives none, : Failure to address contributing factors 1st potted plant thought experiment and both groups are placed in a dark cupboard, Erroneous conclusion: Watering plants is an ineffective intervention the treatment group does not have better outcomes than the control. If 200 potted plants are randomly assigned to either a treatment group that receives daily water, or to a control that receives none, and both groups receive light, Inappropriate question: What proportion of survival and growth is due to the water? the treatment group has better outcomes than the control. : Failure to address combined attribution 2nd potted plant thought experiment 28 Process influence: Goal displacing diverts efforts from achieving goals to meeting targets Gaming and data corruption leads to cynicism (process influence) as well as incorrect decisions (instrumental use) Shaming, reducing peoples mana encourages defensive routines and disengagement Overwhelming scale of deficiencies reduces motivation to try Dissing (disrespecting) inappropriately damages reputation, reducing support Opportunity cost using resources that could be used for operations and for useful evaluation Ways that evaluation can and does have a negative influence Negative influence 29 Gaming High standards 'rob' Victoria Dan Harrison Education Correspondent June 28, 2011 Federal Schools Minister Peter Garrett, who will announce the payments today, said the result reflected the fact Victoria had set ambitious targets. ''These results reflect the fact that Victoria was starting from a high base and was monitoring performances for the largest number of students of all the states, so may have been over-ambitious in its targets.'' STATETARGETPERFORMANCEPAYMENT QLD81.5%88.9%$48.5 MILLION VIC92.98%91.5%$ 9.4 MILLION 30 Some strategies for sharing knowledge about the nightmares of no influence and negative influence Genuine Evaluation With Jane Davidson and guest bloggers Commentary on current examples of genuine and non-genuine evaluation including barriers to influence and risks of negative influence 31 Some strategies for sharing knowledge about the nightmares of no influence and negative influence BetterEvaluationImproving evaluation practice and theory by sharing information about evaluation methods and approaches Founding partners:Overseas Development Institute, Pact, ILAC (Institutional Learning and Change Initiative of the Consultative Group on International Agricultural Research), RMIT University Current funders: Rockefeller Foundation and IFAD (International Fund for Agricultural Development) 32 WEBSITE COMMUNITY Examples Descriptions Tools Guides Comments R & D Documenting Sharing Events 33 34 Sources of content descriptions, examples, guides 35 Addressing complicated aspects of interventions Support evaluation users to translate findings to a new situation Maybe the same theory of change but a different theory of action Understand which contexts enable/prevent the mechanisms Recognise the multiple contributors to impacts Does it have to be pizza? 36 Addressing complex aspects of interventions Support evaluation users in short run learning cycles and ongoing adaptation Support effective working relationships 37 Traditional software development Traditional evaluation development SpecificationTerms of Reference/Evaluation Brief DesignEvaluation Design/Plan Implementation Delivery 38 Agile software development adaptive and responsive as specs change Agile ongoing evaluative inquiry Overall goal Quick buildInitial data sweep (Bron McDonald) Reality testing (Michael Patton) Review and reviseIterative, ongoing inquiry Real time data, self monitoring Repeat quick build 39 Example of real time, collaborative mapping - Ushadishi 40