How to Evaluate Complex Interventions

How to evaluate complex interventions

Astrid Brousselle, Ph.D.

Canada research Chair in Evaluation and Health System Improvement (crc-easy.ca)

Full professor, community health sciences departement

Researcher, Centre de recherche de l’Hôpital Charles-LeMoyne

Université de Sherbrooke

Toronto, 2015, Program Science Symposium

What is evaluation?

- Determining the merit, worth and value (judgement, several criteria)

- Of an intervention, or of its components (resources, process, results)

- Systematic inquiry

- Final objective: improvement of the intervention, social betterment, accountability, social agenda, action

Merriam-Webster: “to judge the value or condition of (someone or something) in a careful and thoughtful way”

Normative evaluation Evaluative research

Assessment of the results

Assessment of the process

Strategic analysis

Effect analysis

Economic eval.

Production analysis

Implementation analysis

Logic analysis

SEVERAL QUESTIONS

Assessment of the structure

The problem

Objectives

Activities/services

Resources

Effects

Context

Lots of models

from Shadish Cook and Leviton (1991)

PracticePractice

Values

KnowledgeUtilization

Social programming Problem

Pawson & Tilley

Source: Alkin MC and Christie CA (2004) An evaluation theory tree. In: Alkin MC (ed.) Evaluation Roots. Tracing Theorists’ Views and Influences. Thousand Oaks, CA: SAGE, 12–65. ADAPTED from Shadish Cook and Leviton (1991)

The utilization paradise

The lobbying

zone

The knowledge-

driven swamp

Evaluation models and use

CONTANDRIOPOULOS, D., BROUSSELLE, A. (2012) “Evaluation models and evaluation use?” Evaluation: The International Journal of Theory Research and Practice, vol. 18: 61-77.

Complex Intervention

Complex interventions are characterized by :

• Uncertainty, unpredictability• Interdependence among a large number of actors• Adapting and evolving actors• Emergent outcomes created by the connections or

relationships in the system• Non-linearity between inputs and outputs

(Shiell, Hawe, & Gold, 2008, cited in Zimmerman et al. 2012 )

Analysing complex interventions

Reduction trap / complexity trap

Embracing complexity:

• Accepting paradoxes (not contradictions) and dealing with them

• Analyzing the relations, the “in between”• Accepting uncertainty• Using multiple perspectives

(Zimmerman et al. 2012 )

A statement that appears to be

self-contradictory or silly but may include a latent

truth.

A statement that appears to be

self-contradictory or silly but may include a latent

truth.

Evaluation answers to interventions’ complexity

Logic analysis

Realistic evaluation

Contribution analysis

Economic value of programs

Logic analysis

Source: Rey et al. 2012

Crucial characteristics of the intervention

Effects

Desired effectsDesired effects

Other effects

Critical conditions

Direct logic analysis

Alternatives (interventions)

Effects

Other effects

Critical conditions

Reverse logic analysis

Desired effectsDesired effects

Realistic evaluation

“What works, for whom, under which circumstances?”

Theory

Hypotheses

Observations

Program specification

MechanismsContextsOutcomes

Contribution analysis

Source: Mayne, 2012

Economic value

TCHOUAKET, E., BROUSSELLE, A., FANSI, A., DIONNE, P-A., BERTRAND, E., FORTIN, C. (2013) The economic value of Quebec's water fluoridation program, Journal of Public Health, 26 juin 2013, open access: springerlink.com

What these approaches share

Centrality of the “theory of change”

Importance of the context

Integration of various perspectives

Plausible interpretation (uncertainty)

Valid and rigorous research process

Creativity

http://www.jr-art.net

ReferencesAlkin MC and Christie CA (2004) An evaluation theory tree. In: Alkin MC (ed.) Evaluation Roots. Tracing Theorists’ Views and Influences. Thousand Oaks, CA: SAGE, 12–65.

Champagne F., Contandriopoulos A-P, Brousselle A., Hartz Z., Denis J-L. (2011-2e édition) « L'évaluation dans le domaine de la santé : concepts et méthodes », dans BROUSSELLE A., Champagne F., Contandriopoulos A‑P., Hartz Z. (Eds) L'Évaluation : Concepts et méthodes, Presses de l’Université de Montréal : 49-70.

Contandriopoulos, D., Brousselle, A. (2012) “Evaluation models and evaluation use?” Evaluation: The International Journal of Theory Research and Practice, vol. 18: 61-77.

Mayne J. Contribution analysis: Coming of age? Evaluation 2012; 18(3): 270-280.

Dubois N, Lloyd S, Houle J, Mercier C, Brousselle A, Rey L. Discussion: Practice-based evaluation as a response to address intervention complexity. Canadian Journal of Program Evaluation 2012; 26(3): 105-113.

Pawson, R., & Tilley, N. (2005). Realistic evaluation. In S. Mathison (Ed.), Encyclopedia of evaluation (pp. 362-367). Thousand Oaks, CA: Sage Publications.

Rey L., Brousselle , A., Dedobbeleer, N. (2012) «Logic analysis: testing program theory to better evaluate complex interventions», dans Houle, J., Dubois, N., Lloyd, S., Mercier C., Hartz, Z., Brousselle, A. (Ed.) (2012) « L'évaluation des interventions complexes », numéro spécial, Revue Canadienne d'Évaluation de Programme/ Canadian Journal of Program Evaluation, Volume 26(3): 61-89.

Shadish W.R., Cook T.D., Leviton L.C. (1991) «Good Theory for Social Program Evaluation», in Shadish W.R., Cook T.D., Leviton L.C., Foundations of program evaluation. Theories of practice, Sage Publications: 36-67.

Shiell, A., Hawe, P., & Gold, L. (2008). Complex interventions or complex systems? Implications for health economic evaluation. BMJ, 336(7656), 1281–1283. doi:10.1136/bmj.39569.510521.AD

Tchouaket, E., Brousselle , A., Fansi, A., Dionne, P-A., Bertrand, E., Fortin, C. (2013) The economic value of Quebec's water fluoridation program, Journal of Public Health, 26 juin 2013, open access: springerlink.com

Zimmerman, B., Dubois, N., Houle, N., Lloyd, S., Mercier, C., Brousselle, A. , Rey, L. (2012) «How does complexity impact evaluation?:  An Introduction to the Special Issue», numéro spécial, Revue Canadienne d'Évaluation de Programme/ Canadian Journal of Program Evaluation, Volume 26(3):v-xx.

[email protected]