Clear Starting Points• English evaluation leader Ray Pawson quotes sociologist Kurt Lewin that
“there’s nothing so practical as good theory”• Pawson wasn’t arguing for abstract theory, but for ensuring we are
always clear about • the assumptions we are making that underpin our work – whether planning a
specific service initiative or developing a broad multi-sectoral strategy• the pathways and factors that we assume will lead from the planning through
service delivery to the hoped-for impact• how we think all of this will vary depending upon the organizational, social or
policy context• there are many approaches and as many terms – theory of change,
program theory, framework for change• but the basic idea is to be really clear about starting points and premises
when planning any initiative • and it’s this set of assumptions, pathways and objectives in our theory of
change that we evaluate
2April 18, 2023 | www.wellesleyinstitute.com
The Problem to Solve = Systemic Disparities in Ontario
•there is a clear gradient in health in which people with lower income, education or other indicators of social inequality and exclusion tend to have poorer health + major differences between women and men•the gap between the health of the best off and most disadvantaged can be huge – and damaging•in addition, there are systemic disparities in access to and quality of care within the healthcare system
3April 18, 2023
‘Wicked’ Policy Problems• health inequities and their underlying social determinants of health are classic
‘wicked’ policy problems:• shaped by many inter-related and inter-dependent factors • in constantly changing social, economic, community and policy environments• action has to be taken at multiple levels -- by many levels of government,
service providers, other stakeholders and communities• solutions are not always clear and policy agreement can be difficult to achieve• effects take years to show up – far beyond any electoral cycle
• have to be able to understand and navigate this complexity to develop solutions • need comprehensive strategy to tackle the roots of health inequality in the
wider social determinants of health• from high-level national social and policy change to reduce inequality through
community-based innovation, cross-sectoral collaboration and mobilization
4April 18, 2023
…Need Theory• for complex social problems – from health inequities, through poverty
reduction, to neighbourhood renewal and comprehensive community initiatives -- we need to be able to:• identify the connections and causal pathways between multiple factors • articulate the mechanisms that we assume drive change in these factors and
the problem as a whole• within that, identify the crucial leverage points that will drive significant
change in the problem• develop strategies for moving these leverage points• specify the short, intermediate and long-term outcomes expected and the
preconditions for achieving them.
that is our theory of change
5April 18, 2023
… And Here’s Howhow to develop theory of change:
• basic principle is to pull group working on the issue together to answer those questions
• to clearly articulate our starting points – what we think are the key causal pathways, mechanisms for change and crucial leverage points
• these are our assumptions and goals for how we think program, community initiative or policy direction will work
• based upon best available evidence, learning and experienceand some wrinkles
• can’t have all the answers and can’t be too prescriptive – don’t wait for ‘perfect’ theory
• be as participatory as possible – especially if community-based
evaluation then tests these assumptions about change mechanisms and outcomes → contributes to further learning and development of the initiative
6April 18, 2023
Theories of Change for ‘Wicked’ Problem I: Health Inequities
complex multi-level strategies to tackle health gaps:
• international frameworks such as WHO Commission on Social Determents
• European Union and other international initiatives
• many individual countrieskey features in all:
• focus on inclusive labour market, anti-discrimination, childcare, affordable housing, social security and other macro policies
• targets and deliverables for relevant departments
• equitable access to quality health care is just one part of this broader package
• emphasized partnerships with community service providers and organizations – in both policy development and service delivery
Theory of change = all about reducing structured inequality :
• health inequities arise out of wider social determinants of health
• won’t solve through health reform alone• underlying structures of social and
economic inequality need to be addressed by new policies
• policies need to be aligned with the incentives and processes that drive government
• key levers will vary – e.g. could be reducing prevalence and impact of precarious employment
• has to involve collaboration and coordination across governments and with many community and non-government stakeholders
7April 18, 2023
Theories of Change for ‘Wicked’ Problem II: Community Level Action
Comprehensive community initiatives:• broad partnerships of local residents,
community organizations, governments, business, labour and other stakeholders
• coming together to address deep-rooted local problems – poverty, neighbourhood deterioration, health inequities, racism
• collaborative cross-sectoral efforts – not just for better coordinated service provision but for community development and capacity building
• Vibrant Communities – 14 communities across the country working to build individual and community capacities to reduce poverty
Theory of change:• poverty, social exclusion, etc are not
individual problems but are deeply rooted in fundamental social and economic inequality → even the best programs cannot get at these underlying structures
• underlying structures of inequality need to be addressed by new policy directions
• building community capacities is another direction to address structural inequalities:• drilling down – Vibrant Communities
works to build up individual skills and other assets as building block for community change
• broad collaborations are key to both enhancing community capacities and implementing policy changes locally
• broad coalitions also can highlight the deep-seated nature of problems → build community and public awareness and focus political advocacy
8April 18, 2023
Evaluating Complex Policy and Community Initiatives
• far too complex to pick apart all the causal relations and patterns of influence:• very difficult to attribute particular changes to particular components of the overall
initiative• will never meet RCT gold standard of proof – that approach can’t capture complexity• but that doesn't mean initiative is ineffective
• impact can take many years to show up• and that doesn’t mean nothing is happening
• traditional evaluation of one program in isolation or of a particular population among many will not capture this complexity
• so we evaluate our theory of change• Tamarack talks of framework for change – to highlight the breadth of activities and
stakeholders involved in comprehensive initiatives• we identified levers in our strategy – did they prove to be important in practice?• looking for indications that the change mechanisms unfold as we expected, that the
direction of causal influence and impact is as we expected ,etc• looking for evidence that outcomes anticipated are being achieved, and the change
mechanisms we anticipated have been important
9April 18, 2023
Build Use and Users Into Evaluation
• crucial to identify users of evaluation • what they need • the context in which they will interpret and use the results of evaluation
• policy makers need to:• solve social problems as they see them• determine where to invest scarce resources for maximum impact• avoid mixing up accountability requirements with evaluation• get beyond the assumption that randomly controlled trials are the ‘gold
standard’ of evidence• community leaders and members are interested in:
• the same – how to solve social problems as they see them – and live them• how to make case for reform/spending to policy makers• how to ensure programs and services meet the needs of community members
– and how to continually improve service delivery• both need to be able to determine – and demonstrate – what types of
initiatives work, for whom and in what contexts
10April 18, 2023
Case Study: Health Equity Strategy Into Action
• goal is to ensure equitable access to high quality healthcare regardless of social position
• can do this through a multi-pronged strategy:1. building health equity into all health care planning and delivery
• doesn’t mean all programs are all about equity• but all take equity into account in planning their services and outreach
2. aligning equity with system drivers and priorities3. embedding equity in provider organizations’ deliverables, incentives
and performance management 4. targeting some resources or programs specifically to addressing
disadvantaged populations or key access barriers• looking for investments and interventions that will have the highest impact
on reducing health disparities or enhancing the opportunities for good health of the most vulnerable
5. while thinking up-stream to health promotion and addressing the underlying determinants of health
11April 18, 2023
Equity-Focused Planning• a critical component of all these strategic directions is good planning• addressing health inequities requires a solid understanding of:
• key barriers to equitable access to high quality care• the specific needs of health-disadvantaged populations• gaps in available services for these populations
• great deal of emphasis on tools such as Health Equity Impact Assessment
• but can’t see particular tool as ‘magic bullet• to develop effective planning, we need:
• clear strategy• a coherent approach• a repertoire of effective tools and techniques• support for planners and practitioners to effectively use them• good actionable information
• and drilling down: what is our ‘theory’ of how equity-focused planning works?
12April 18, 2023
13April 18, 2023
taking account of social
constraints & conditions
not just individual
programs but coordination,
partnerships & collaboration
14April 18, 2023
enhanced access to
health promotion for
most disadvantaged
up-stream heath conditions & opportunities
improve fastest for those in
greatest need
Complexities At All Stages• processes and constraints are complex, and outcomes
uncertain and unpredictable, at each of these junctures• and all of this varies by context:
• particular communities or neighbourhoods – with their different health challenges and needs
• particular population health and service landscape in specific areas• community resilience, connectedness, organizing and traditions• further specified by health condition or concern (e.g. the determinants
and dynamics of mental health are different)• we need to know what works best at each of these junctures
(and cumulatively), what programs and frameworks work for particular populations, and what works in varying contexts
→ need to build evaluation in from the start to learn
15April 18, 2023