Developing Developing evaluation evaluation frameworks for frameworks for large, multi- large, multi- program projects: program projects: Building logic “meta- Building logic “meta- models” and visual models” and visual databases databases Jason Newberry , Andrew Taylor, Robert Case, and Kristen Roderick Centre for Research and Education in Human Services [email protected]
39
Embed
ABSTRACT Developing evaluation frameworks for large, multi-program projects: Building logic “meta-models” and visual databases Authors: Jason Newberry,
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Developing evaluation Developing evaluation frameworks for large, frameworks for large,
multi-program multi-program projects:projects:
Building logic “meta-models” and Building logic “meta-models” and visual databasesvisual databases
Jason Newberry, Andrew Taylor, Robert Case, and Kristen Roderick Centre for Research and Education in Human Services
How large government policy frameworks How large government policy frameworks create new challenges for health and create new challenges for health and social service evaluators social service evaluators
Ontario’s Early Childhood Development Ontario’s Early Childhood Development initiatives: A Work in Progressinitiatives: A Work in Progress
The development of an evaluation The development of an evaluation framework: logic “meta-models” and visual framework: logic “meta-models” and visual databasesdatabases
Please Note:Please Note:
Although this presentation draws on Although this presentation draws on examples from our ongoing work with examples from our ongoing work with programs run by the Ontario government, programs run by the Ontario government, the specific content of these examples is the specific content of these examples is intended only to illustrate the evaluation intended only to illustrate the evaluation process we have designed and does not process we have designed and does not reflect the official position of the Ontario reflect the official position of the Ontario Government regarding these programs.Government regarding these programs.
A New Reality for EvaluatorsA New Reality for Evaluators In the public sector, there is a greater need for evaluation In the public sector, there is a greater need for evaluation
because of:because of: A rebirth of interest in the social determinants of health and A rebirth of interest in the social determinants of health and
primary prevention.primary prevention. A new emphasis on results and evidence-based management.A new emphasis on results and evidence-based management. A new acknowledgement of the need to let different regions A new acknowledgement of the need to let different regions
develop unique approaches to addressing shared priorities.develop unique approaches to addressing shared priorities. Evaluators are now confronted with the task of Evaluators are now confronted with the task of
evaluating the impact of many diverse programs that fall evaluating the impact of many diverse programs that fall under very broad funding strategies. under very broad funding strategies.
This requires developing evaluation frameworks and This requires developing evaluation frameworks and strategies that are capable of evaluating very different strategies that are capable of evaluating very different programs that have been created to achieve similar programs that have been created to achieve similar goals.goals.
A Specific Example: A Specific Example: Early Childhood Development Early Childhood Development
Initiative in OntarioInitiative in Ontario Federal/Provincial funding of programs that broadly target early Federal/Provincial funding of programs that broadly target early
childhood development (0-6 years) and encourage different childhood development (0-6 years) and encourage different jurisdictions to develop their own approachesjurisdictions to develop their own approaches
In Ontario, 26 different provincial initiatives are fundedIn Ontario, 26 different provincial initiatives are funded Most initiatives have multiple service sites (e.g., one initiative has Most initiatives have multiple service sites (e.g., one initiative has
over 100 program sites)over 100 program sites) Within initiatives, each site may offer different services, or have a Within initiatives, each site may offer different services, or have a
different focusdifferent focus Types of interventions are very diverse. For example:Types of interventions are very diverse. For example:
Direct clinical treatment of childrenDirect clinical treatment of children Parent education and skill developmentParent education and skill development Pre- & post-natal servicesPre- & post-natal services Promotional campaigns, awareness-raisingPromotional campaigns, awareness-raising Social and practical supportSocial and practical support
A new emphasis on public sector A new emphasis on public sector accountability may lead to questions accountability may lead to questions like….like….
““Are funded Early Years programs in the Are funded Early Years programs in the province effective in improving early province effective in improving early childhood development?”childhood development?”
Beginning to answer “meta- questions”
New frameworks are needed that can lead to… ways of speaking to meta-level evaluation questions
like “to what degree did diverse, multi-site programs function as components in a larger coordinated plan?”
ways of comparing and consolidating the theories of change underlying diverse interventions with similar goals.
ways of comparing and consolidating the measurement approaches to diverse interventions.
What are the barriers?What are the barriers? Information overload and difficulty in organizing Information overload and difficulty in organizing
that information.that information. Information “underload” where evaluators are Information “underload” where evaluators are
twice and thrice removed from the field.twice and thrice removed from the field. Neither of the two “traditional responses” works:Neither of the two “traditional responses” works:
Large, centralized, standardized evaluation systems Large, centralized, standardized evaluation systems do not transfer well to the field; exceedingly difficult to do not transfer well to the field; exceedingly difficult to design with diverse programs.design with diverse programs.
Decentralized approaches which encourage each Decentralized approaches which encourage each program or community to do their own evaluation program or community to do their own evaluation often may promote poor quality evaluation designs; often may promote poor quality evaluation designs; makes very difficult to be summative across makes very difficult to be summative across programs.programs.
What are the solutions?What are the solutions?
Blended approaches, which emphasize Blended approaches, which emphasize shared theory of change and shared shared theory of change and shared definitions of key evaluation terms, but definitions of key evaluation terms, but allow for program-to-program and region-allow for program-to-program and region-to-region diversity in how measurement is to-region diversity in how measurement is carried out.carried out.
Using available technology to facilitate the Using available technology to facilitate the creation, understanding and ongoing creation, understanding and ongoing coherence of evaluation frameworks.coherence of evaluation frameworks.
Four broad target areas:Four broad target areas: promote healthy pregnancy, birth and infancy;promote healthy pregnancy, birth and infancy; improve parenting and family supports;improve parenting and family supports; strengthen early childhood development, strengthen early childhood development,
learning and care; and learning and care; and strengthen community supports.strengthen community supports.
• Knowledge
• Skill
• Use of Services
• Parenting Behaviour
• Support
• Independence
• Use of Services
• Stress
• Early ID
• Behaviour
• Readiness
• Knowledge
• Collaboration
• Consistency
• Capacity
• Access
• Knowledge
• Awareness
• Commitment
• Support
Child Health
Family
Health
Community
Health
Service
System
• Parent Education
• Personal Support
• Assessment & Counselling
• Interactive Learning
• Clinical Care, Nutrition
• Screening & Assessment
• Referral
• Resources and Tools
• Networking
• Training
• Program Effectiveness
• Public Education
• Community Advocacy & Leadership
The Main Activities and Outcome Objectives of Early Childhood Development Initiatives
Examples of Preventive Activities and Associated Outcome Objectives in Childhood Development Initiatives
Child Health
Family
Health
Community
Health
Service
System
• Personal Support
• Assessment & Counselling
• Screening & Assessment
• Referral
• Use of Services
• Parenting Behaviour
• Support
• Use of Services
• Early ID
• Behaviour
• Readiness
Example of Assessment and Treatment Activities and Associated Outcome Objectives in Early Childhood Development Initiatives
Child Health
Family
Health
Community
Health
Service
System
• Parent Education
• Knowledge
• Skill
• Use of Services
• Parenting Behaviour
• Stress
• Early ID
• Behaviour
• Readiness
Activities often have Outcomes at Multiple Levels
Implications for Practice Implications for Practice
Evidence Based
PlanningVision or Plan
Priority Goal Priority Goal
Outcome Evaluation
Activity or Program
Activity or Program
Activity or Program
Short-TermOutcome Objective
Short-Term Outcome Objective
Long-TermOutcome Objective
Long-TermOutcome Objective
Long-TermOutcome Objective
Beginning with Program MapsBeginning with Program Maps Initiative Focus Pregnancy, Birth and InfancyInitiative Components Direct Services
Activity Categories
Providing parent education & training (healthy pregnancy, prenatal nutrition, parenting transition, childbirth information, father involvement, etc.)
Providing screening & assessment Providing counseling Providing professionally led peer support groups
(e.g., prenatal topics) Providing personal & practical support (e.g,
lactation aids, baby loan depot) Providing referral information & hotline
Short-termOutcome Objectives
Improved knowledge of supports & services for healthy pregnancy & infant development
Increased use of other supports & services Improved parenting knowledge & practices Improved identification of at-risk parents, infants Increased feelings of family & social support
Long-Term Outcome Objectives
Increased parenting capacity Improved maternal health Improved infant health & well-being
Beginning with Program MapsBeginning with Program Maps
Promote children’s optimal development and readiness to learn, so that they may
reach their fullest potential
Improved child health &
physical well-being
AC
TIV
ITIE
S
SH
OR
T-
TE
RM
O
BJE
CT
IVE
S
GO
AL
S
LO
NG
-TE
RM
O
BJE
CT
IVE
S
Sample Program Logic Model
Program and System Supports
Developing & implementing
service coordination
policies, protocols
For all children to be healthy, safe, and secure within strong families and
communities
Pregnancy, Birth, and Infancy --- Parenting & Family Support
Increased accessibility of
pregnancy & child development intervention
programming
Improved school
readiness
Increased child
prosocial behaviours
Increased coordination & integration of services in community
Increased entry into appropriate
supports & services
Creating, disseminating
child development
resource materials
Developing networks of
service providers
Training of, consulting
with, service providers
Increased service provider
knowledge of pregnancy & child
development supports &
services/gaps
Increased service provider, parent, &
community awareness of the importance of the
early years
Increased parent &
community awareness of pregnancy &
child development
services
Promotional advocacy
campaigns
Ongoing parental
awareness media
campaigns
Organizing community
events
Improved knowledge of supports & services for
healthy pregnancy & infant/child
development
Providing referral info & hotline
Increased use of other supports &
services
Improved parenting
knowledge and practices (pre- & postnatal care, nutrition, child development,
parenting skills, health/safety)
Increased service provider
collaboration
Increased general
awareness of the importance
of parenting
Increased commitment,
support, programming
for EY services
Increased community
participation in EY &
prevention activities
Improved identification of at-risk parents,
infants and children (PPD,
P4, developmental
delay, etc).
Improved maternal
health
Improved infant health & well-being
Promote Healthy Pregnancy & the Birth of Healthy Babies
Provide small
learning groups for children
Providing parent
education & training
Providing professionally
led support groups for
parents
Providing child
screening & assessment
Providing counseling to parents
Providing personal, practical
support to parents
Evaluating provincial NP
services & community
impact
Increased capacity to
assess program
planning & effectiveness
Enhancing & expanding
core programming
Increased SP knowledge & skills
of pregnancy & child development best practices &
interventions
Increased child
stimulation &
interaction
Early Childhood Development,
Learning & Care
Providing parent
screening & assessment
Link to Map
Increased feelings of family & social
support
Increased parenting capacity
Improved emotional
functioning & well being of
children
Short-term Outcome Objectives
Improved knowledge of supports and services available
Improved knowledge of supports & services for healthy pregnancy & infant development
Increased use of supports & services Increased use of other supports & servicesIncreased parenting knowledge & practical skills Improved parenting knowledge & practicesIncreased early diagnosis & intervention Improved identification of at-risk parents, infantsIncreased social opportunities, networks & feelings
of support Increased feelings of family & social support
Coding Initiative MapsCoding Initiative Maps
Short-term Outcome Objectives
Improved knowledge of supports and services availableImproved knowledge of supports & services for healthy pregnancy & infant developmentIncreased use of supports & services Increased use of other supports & servicesIncreased parenting knowledge & practical skills Improved parenting knowledge & practicesIncreased early diagnosis & interventionImproved identification of at-risk parents, infants Increased social opportunities, networks & feelings of supportIncreased feelings of family & social support
Coding Initiative MapsCoding Initiative Maps
We can examine which initiatives share common activities and the different service approaches used...
Drop-insDrop-ins Home visits Home visits & outreach& outreach
Activity Activity CategoriesCategories
PROVIDING INTERACTIVE LEARNING, ARTS, AND PROVIDING INTERACTIVE LEARNING, ARTS, AND PLAY OPPORTUNITIES PLAY OPPORTUNITIES Areas of focusAreas of focus: literacy, numeracy, homework clubs, arts & : literacy, numeracy, homework clubs, arts & crafts, play-based problem solving, empathy crafts, play-based problem solving, empathy
IMPROVED EMOTIONAL FUNCTIONING & WP21L-BEING OF CHILDREN
P3, P7, P5, P4, P9, P8
IMPROVED COGNITIVE ABILITY
P2, P6, P3, P7, P21, P9, P11, P8, P9, P16, P17
IMPROVED SOCIAL SKILLS, COMMUNICATION, & BEHAVIOUR
P2, P6, P3, P7, P21, P9, P11, P8, P9, P16, P17
Long-term
outcome objectives
MISCP21LANEOUS INITIATIVE & SYSTEM SPECIFIC OBJECTIVES (e.g., increased early disclosure of sexual assault, greater immunization rates, decreases FAS incidence & infant morality, greater system independence)
P2, P7, P8, P15, P16, P17
We can examine which initiatives share common short-term and long-term objectives
IMPROVED EMOTIONAL FUNCTIONING & WELL-BEING OF CHILDREN
P3, P7, P4, P4, P9, P8
IMPROVED COGNITIVE ABILITY
P2, P6, P3, P7, P21, P9, P11, P8, P10, P16, P17
IMPROVED SOCIAL SKILLS, COMMUNICATION, & BEHAVIOUR
P2, P6, P3, P7, P21, P9, P11, P8, P10, P16, P17
Long-term outcome
objectives
MISCELLANEOUS INITIATIVE & SYSTEM SPECIFIC OBJECTIVES (e.g., increased early disclosure of sexual assault, greater immunization rates, decreases FAS incidence & infant morality, greater system independence)
P2, P7, P8, P15, P16, P17
We can move from the roll-up to the unique logic models of particular initiatives. These contain customized versions of shared objectives in the roll-up (in this example, P6)
Providing parenting services (information,
education and support)
Improved knowledge
of avail. supports& services
Increased parenting
knowledge and skill
Increased use of
services and supports
Increased parenting capacity
Providing a variety of targeted
programs for special
populations, including
teen parents
Providing ethnically and
culturally appropriate programs
Providing school
readiness programs
Providing programs which
support children’s healthy
development
Providing funding for community
based projects that meet identified
criteria
Providing ongoing support for
funded projects, and support for proposal development
Supporting local planning
and proposal review
Creating and running awareness campaigns
Creating community resources
Engaging local non-traditional
partners aschampions, funders
and committee members (e.g.,
business)
Improved child health and
physical well-being
Improved language and
cognitive skills of children
Improved social skills/competency
of children
Improved social maturity of
children
Improved communication
skills of children
Increased awareness of
avail community services
Increased community
participation in early years activities
Increased community collaboration/linkages
among service providers and other
supporters of parents
Increased new and locally-
created programs to fill gaps
Increased fit between (new and existing) services
and local priorities and
identified needs
Increased knowledge
among service partners about local gaps and
prioritiesIncreased
awareness in the community at large of the
importance of EY
Increased general awareness of the
importance of parenting
Improved sustainability of
services
Increased diversity of funding and
supports
Improved program quality
Decreased duplication of
services
More dense network of inter-
relationships among EY
services and other family supports
Increased commitment/supp
ort for EY services
Increased use of
services and
supports
Promote children’s optimal development
and readiness to learn, so that they may reach their fullest potential
Increased access to a more cohesive,
comprehensive range of EY programs,
services and parenting supports
Improved quality and capacity of existing children’s services
Stronger sense of shared philosophy for young children and
their families
Prenatal and Postnatal
(information, education and
support)
Increased opps. for children to participate in activities that
promote healthy development
Increased access to services for
targeted populations
Increased size and density of
families’ social
networks
Improved/smoother transitions to school
Increased linkages between service partners and alternative
resources
Reduced parent stress
Reduced child stress
Activities
Short term objectives
Long term
objectives
EY goals P6 goals
More dense network of inter-
relationships among EY
services and other family supports
Providing parenting services (information,
education and support)
Improved knowledge
of avail. supports& services
Increased parenting
knowledge and skill
Increased use of
services and supports
Increased parenting capacity
Providing a variety of targeted
programs for special
populations, including
teen parents
Providing ethnically and
culturally appropriate programs
Providing school
readiness programs
Providing programs which
support children’s healthy
development
Providing funding for community
based projects that meet identified
criteria
Providing ongoing support for
funded projects, and support for proposal development
Supporting local planning
and proposal review
Creating and running awareness campaigns
Creating community resources
Engaging local non-traditional
partners aschampions, funders
and committee members (e.g.,
business)
Improved child health and
physical well-being
Improved language and
cognitive skills of children
Improved social maturity
of children
Improved communication
skills of children
Increased awareness of
avail community services
Increased community
participation in early years activities
Increased community collaboration/linkages
among service providers and other
supporters of parents
Increased new and locally-
created programs to fill gaps
Increased fit between (new and existing) services
and local priorities and
identified needs
Increased knowledge
among service partners about local gaps and
prioritiesIncreased
awareness in the community at large of the
importance of EY
Increased general awareness of the
importance of parenting
Improved sustainability of
services
Increased diversity of funding and
supports
Improved program quality
Decreased duplication of
services
Increased commitment/supp
ort for EY services
Increased use of
services and
supports
Promote children’s optimal development
and readiness to learn, so that they may reach their fullest potential
Increased access to a more cohesive,
comprehensive range of EY programs,
services and parenting supports
Improved quality and capacity of existing children’s services
Stronger sense of shared philosophy for young children and
their families
Prenatal and Postnatal
(information, education and
support)
Increased opps. for children to participate in activities that
promote healthy development
Increased access to services for
targeted populations
Increased size and density of
families’ social
networks
Improved/smoother transitions to school
Increased linkages between service partners and alternative
resources
Reduced parent stress
Reduced child stress
Activities
Short term objectives
Long term
objectives
EY goals P6 goals
Improved social skills/competency
of children
has led to clearer and more accurate framework and terminology for summarizing early years activities and objectives.
has allowed us to describe activities in terms of both content and service delivery modality.
The ECD Visual Data BaseThe ECD Visual Data Base
allows us to talk about the focus of various types of work within the Early Years plan, not just presence and absence.
helps us to work towards cross-initiative consistency in understanding important outcome objectives and how they relate to one another.
The ECD Visual Data BaseThe ECD Visual Data Base
allows ministry staff to quickly and easily understand individual initiatives and the connections between them.
contributes to a better understanding of how initiatives fit into the 4 priority areas the Federal/Provincial agreement.
The ECD Visual Data BaseThe ECD Visual Data Base
Moving to measurementMoving to measurement
We have recently entered a phase that We have recently entered a phase that moves the framework towards moves the framework towards measurementmeasurement
We are repeating a process that is similar We are repeating a process that is similar to the generation of the program maps.to the generation of the program maps.
Each initiative is completing a table of Each initiative is completing a table of indicators, data items & sources, and indicators, data items & sources, and design/analysis recommendations, that design/analysis recommendations, that are linked to each objective are linked to each objective
Moving to measurementMoving to measurement
These tables will also be electronically These tables will also be electronically linked to the visual database (the roll-up linked to the visual database (the roll-up table and logic models)table and logic models)
ROLL UP OF ECDI INITIATIVES
Initiative Focus
Pregnancy, Birth, and Infancy
IMPROVED KNOWLEDGE OF SUPPORTS AND SERVICES AVAILABLE P9, P8, P3, P4, P14, P10, P18, P15, P16
INCREASED USE OF SUPPORTS AND SERVICES P1 P3, P4, P9, P14, P15, P10, P18, P16, P15 INCREASED PARENTING KNOWLEDGE AND PRACTICAL SKILLS:
Developmentally appropriate activities for children P23
ages and stages of child development P1 P13, P4, P8, P10, P23, P18
increased breastfeeding P1 P13, P10, P16, P18
child health and safety P1 P13, P4, P15, P16, P18
parent/child interaction P8, P10
behaviour management P8
language and cognitive development P8
child immunization P23
healthy lifestyle choices P3, P4
preventive dental health practices during pregnancy and early childhood P23
impact of addictions on pregnant women and their babies P4, P16
FAS/FAE (effects of alcohol/drugs, risk factors, and related issues) P4, P16 parental coping skills P10 INCREASED EARLY DIAGNOSIS AND INTERVENTION P3, P4, P10
Short-term
outcome objectives
INCREASED SOCIAL OPPORTUNITIES AND SOCIAL NETWORKS P16, P18
For a given initiative, we can move from a logic model to the roll-up, and from there to the indicator table to examine specific evaluation questions, methods, and measures. These may be transferable to initiatives sharing common objectives...
Outcome Objective Monitoring and Evaluation Questions
q1. What health & wellness behaviours were women more aware of and changes made post-partum re: safety nutrition, healthy growth, development? (H-N)
Women’s self-reports of their knowledge of healthy behaviours (H-N)Women’s self-reports of their health-related behaviours (H-N, PER)
Women’s self-reports of their knowledge & skills re: early child care & development(PER)
Clients’ self-reports of their knowledge of early childhood developmental milestonesorClients’ level of knowledge of early childhood developmental milestones(SUD)
Women’s self-reports of choosing healthy lifestyle behaviours during pregnancy (E-ST, H-N, SUD)
Table of Indicators, Measurement, and Design
CurrentDataitem(s)
Current data collection &managementmechanism(s)
Analysis plan for current data, gaps
Recommendations, feasibility, importance, gaps
Responses on “Client Behaviour Survey” (H-N)
Data gathered by NPs, in a pre-post design (H-N)
(no measure for PER)
Change may not be relatively uniform if services vary
Need details on ClientBehaviour Survey
Questionnaire responses
Ongoing data collection by NPs
Is this pre-post?
Two indicators provided here b/c it is unclear if the data item gathers self-reports of knowledge, or content (test-based) knowledge.
Table of Indicators, Measurement, and Design
The Indicator libraryThe Indicator library
This current phase of developing indicators, This current phase of developing indicators, examining current evaluation data, and examining current evaluation data, and proposing future evaluation activities has led to proposing future evaluation activities has led to the development of an “indicator library”.the development of an “indicator library”.
The indicator library is analogous to our The indicator library is analogous to our development of the roll up. While the roll-up development of the roll up. While the roll-up summarized ECD activities and objectives, the summarized ECD activities and objectives, the indicator library summarizes measurement and indicator library summarizes measurement and design across all the initiatives.design across all the initiatives.
facilitates cross-site collaboration within facilitates cross-site collaboration within initiatives but also collaboration across initiatives but also collaboration across initiatives.initiatives.
allows for the possibility of consistency of allows for the possibility of consistency of measurement across programs that share measurement across programs that share common objectives, programs models, common objectives, programs models, and populations.and populations.
fosters best practices in evaluation. fosters best practices in evaluation.
The Indicator libraryThe Indicator library
The lessons learnedThe lessons learned We began by building consistent and concrete We began by building consistent and concrete
theories of changetheories of change across diverse situations, across diverse situations, rather than trying to impose a consistent rather than trying to impose a consistent measurement system.measurement system.
We did this work with local programs and broad We did this work with local programs and broad province-wide systems province-wide systems at the same timeat the same time and and used each level to clarify the other.used each level to clarify the other.
This strategy has allowed us to use an This strategy has allowed us to use an incremental approachincremental approach to improving to improving measurement and reporting practices at many measurement and reporting practices at many levels and over time, which is cost-effective and levels and over time, which is cost-effective and builds ownership. builds ownership.
Overall meta logic models(1 for each priority area)