Mosaic Counselling and Family Services · Mosaic Counselling and Family Services . ... F&ST has used a prepost non- -experimental design with two raters ... issues and agendas to

1

Mosaic Counselling and Family Services

Evaluation Implementation Grant (EIG - 1167)

Program: Families and Schools Together (F&ST)

September, 2010

2

Executive Summary

The Families and Schools Together (F&ST) program was created in 1988 to prevent school

failure, substance abuse, violence and delinquency in “at-risk children” from stressed, isolated

and low-income families by addressing the needs and providing support to families and

promoting community development1 (McDonald, Billingham, Conrad, Morgan & Payton, 1997, p.

141) and has been offered by Mosaic Counselling and Family Services (Mosaic) for nine years.

For the past several years, F&ST has used a pre-post non-experimental design with two raters

(parent and teacher). The standardized evaluation instruments have included several

quantitative methodologies such as questionnaires and an open-ended questionnaire providing

the family an opportunity to explain how the program impacted their lives. Parents and teachers

completed the questionnaires before the program began and after it ended. The measures were

paired to determine the level of change2. It was discovered that many people, including F&ST

staff as well as school personnel did not feel that the information gathered using these data

collection methods was meaningful to them.

The current evaluation involved using a variety of research methods including quantitative

surveys as well as qualitative open-ended questionnaires, interviews, observation and an

innovative technique called Photovoice. Participants involved in the evaluation activities

included F&ST families, parent partners, school personnel, F&ST staff and volunteers, and

personnel from Public Health. Thirty-three interviews were conducted with individuals across all

capacities/roles of the program and eleven individuals participated in Photovoice.

The findings suggest that the F&ST program has many short-term benefits to those involved

and has been shown to accomplish several of the short term goals including connecting families

to their school, allowing them to meet new people within their community, and is enjoyed by

those who participate including F&ST staff, F&ST volunteers, Public Health personnel and

school personnel as well.

1 McDonald, Billingham, Conrad, Morgan & Payton, 1997 (p. 141) 2 Family Service Canada, n.d.

3

However, according to a description of the program components and roles of staff and

volunteers3, there are disconnects between the documentation and actual practice and

distinctions between actual and desired long term outcomes. It may be that in order to achieve

the longer term outcomes of the program and therefore accomplish all of the goals of the

program, the program may have to adhere more closely to the original model4.

There are many aspects of the program that are experienced as valuable and successful by all

of the participants, including time to connect with family, meet new people, connecting with

school and community, and linking to Mosaic Counselling Services. Further, some of the

successful aspects of the program could be built on in order to provide even better services to

clients.

Using a variety of evaluation techniques, new information and insight has been gathered

regarding the successes, challenges as well as short and long term outcome of the program.

Further, the new evaluation process was more enjoyable for participants than earlier methods

used. Of particular success was Photovoice which all families who were involved in enjoyed.

Photovoice was felt to be fun by parents and children and offered people an opportunity to

demonstrate through images what happens in F&ST. Asking questions about the lives and

experiences of families was less intrusive when families could select the pictures and

experiences that they wanted to talk about.

The experiences and learnings generated from the current evaluation are important to share

with the agency as well as others and there are several recommendations for how to amend the

program as well as prepare for and conduct evaluation in the future. The experience of working

with the Centre has been positive for a multitude of reasons.

From an agency perspective, the in-depth nature of this evaluation project is a first. The

qualitative evaluation methods including participant observation and interviews offered a close

look at the inner workings of a Mosaic program. The results are recommendations for aspects of

the program to retain and aspects to change, based on extensive study and review of the

literature.

3 McDonald, Billingham, Conrad, Morgan & Payton, 1997 4 Ibid

4

Table of Contents

EExxeeccuuttiivvee SSuummmmaarryy ................................................................................................................... 2

IInnttrroodduuccttiioonn ................................................................................................................................ 6

History and Purpose of F&ST .............................................................................................. 6

Overview of the Program and Activities ............................................................................. 6

Some Research on F&ST ..................................................................................................... 8

Past F&ST Evaluation Processes at Mosaic .....................................................................10

Purpose of the Current Evaluation ....................................................................................11

MMeetthhooddoollooggyy .............................................................................................................................11

Methods Used in the Current Evaluation ...........................................................................11

Methodology Limitations ....................................................................................................17

RReessuullttss ......................................................................................................................................18

Strengths of the Program ...................................................................................................18

Challenges of the Program .................................................................................................20

Observational Findings ......................................................................................................25

Previous Evaluation of the Program ..................................................................................26

New Evaluation Process .....................................................................................................26

Other Information ................................................................................................................27

DDiissccuussssiioonn aanndd LLeessssoonnss LLeeaarrnneedd..............................................................................................28

Sharing Information ............................................................................................................28

Learning ...............................................................................................................................29

The Evaluation Experience .................................................................................................29

CCoonncclluussiioonnss aanndd RReeccoommmmeennddaattiioonnss//NNeexxtt SStteeppss .......................................................................32

Recommendations for Others ............................................................................................32

Recommendations for the Organization ............................................................................33

KKnnoowwlleeddggee EExxcchhaannggee ...............................................................................................................36

5

Works Cited……………………………………………………………..……………………………....38 AAppppeennddiicceess ...............................................................................................................................39

Appendix A: F&ST Logic Model and Evaluation Framework ................................................40

Appendix B: Informed Consent for F&ST Families and Parent Partners ..............................43

Appendix C: Interview Consent Form for F&ST Staff and Volunteers, School Personnel,

Public Health Personnel and Mosaic F&ST Staff ...................................................................44

Appendix D: Interview Schedules ........................................................................................45

Appendix E: Recruitment Flyer for Potential Photovoice Participants ..................................49

Appendix F: Informed Consents for Photovoice Participants................................................50

Appendix G: Information and PowerPoint Introduction to Photovoice for Families ...............51

Appendix H: Photovoice Focus Group Recruitment Flyer ...................................................58

Appendix I: Photovoice Focus Group Informed Consent Form ............................................59

Appendix J: One Page Summary for Poster Conference at CHEO ......................................60

6

Introduction History and Purpose of F&ST Lynn McDonald created the F&ST program in 1988 to prevent school failure, substance abuse,

violence and delinquency in “at-risk children” from stressed, isolated and low-income families by

addressing the needs and providing support to families and promoting community development

(McDonald, Billingham, Conrad, Morgan & Payton, 1997, p. 141). McDonald recognized that

traditional mental health services including individual or family therapy often failed to reach ‘at-

risk’ clients and also failed to address the overall impact of one’s environmental context. For

example, factors such as unemployment, violence or alienated communities affect the well-

being of children and particularly their ability to thrive in school. Further, these environmental

factors are linked with later substance abuse, violence, delinquency and school failure. The

F&ST program, then, was created to help build families and neighbourhoods, with a goal of

improving individual and social functioning (McDonald et al., 1997).

Social isolation, low income, unemployment, unsafe neighbourhoods and other environmental

factors can be detrimental to parents as they may come to believe that they are unfit to be

parents and undermine their confidence in their capacity to effectively parent their children.

Connecting them with neighbours within their community allows for these parents to create

natural relationships which they can rely on as a source of support rather than relying on

therapeutic assistance. Community connection in addition to stronger familial connections

creates feelings of empowerment for parents and also works to increase child functioning

(McDonald et al., 1997). According to McDonald (1997) “F&ST is not simply a children and

families program that builds on protective factors, but is a community-building strategy as well”

(p.146).

Overview of the Program and Activities

According to the literature, F&ST is designed to be a two-year program consisting of an eight-

week F&ST cycle and a two year F&STWORKS cycle held immediately afterward (Family

Service Canada, n.d.; McDonald, 1997).

Partner Selection Before a F&ST program can be run in any school, a F&ST team including at least four partners

– “a parent from the local community, two community agency professional representatives (one

from a mental health agency and one from a substance-abuse program), and one professional

7

from the local elementary school” (p. 145) – must be formed. All members of this team must

commit to being involved in the program for the full two years (McDonald, et al., 1997).

Training F&ST teams must receive training on how to run the F&ST program (Family Service Canada,

n.d.). F&ST trainers are also responsible for visiting two or three times during the course of the

eight week program. After the program has ended, the trainer is required to meet with F&ST

staff to reflect on their experiences of the program as well as determine what worked well as

well as what did not work as well in that particular community. Trainers also seek feedback from

program participants about the program and work with them to concretize the needs, issues and

agendas to be addressed in F&STWORKS (Family Service Canada, n.d.).

Participant Recruitment Families are primarily recruited to F&ST through their children: ‘at-risk’ children are identified by

their teachers while in elementary school. Once potential child participants are identified, a

“parent-professional team” which usually consists of F&ST graduates formally invites the family

of that child to participate in the program (p. 141). Inviting parents to participate offers them an

opportunity to make a decision/choice and works to increase their level of self-confidence

(McDonald, et al., 1997).

Program Sessions F&ST sessions are identical in their agenda/schedule with each activity carefully structured and

planned based on empirical research: the F&ST program is founded on the family stress theory

as well as “systematically integrates exemplary practices from both community organizing and

clinical casework and uses positive engagement of families and community to help at-risk

children succeed at school, at home, and in their community” (McDonald, 1992 in McDonald, et

al., 1997, p. 141). Sessions are led by F&ST staff members but to assist them, there are several

additional F&ST team members who take part including a parent graduate of the program, a

member of the school, a social worker and a substance abuse counsellor (McDonald, et al.,

1997, p. 144). F&ST staff members are made available to recruited F&ST families to contact

with questions or concerns during the week between F&ST sessions (McDonald, et al., 1997). A

more detailed description of each of the activities engaged in through F&ST and the theory

behind each is referred to in the F&ST Program Workbook (Family Service Canada, n.d.).

8

F&STWORKS After families graduate from the F&ST program, they are invited to attend F&STWORKS

(meetings held monthly for two years to help sustain the skills and knowledge learned in F&ST).

F&STWORKS is designed to assist parents become more independent as well as offer support

to one another. By the time parents begin F&STWORKS, they should already feel more

connected their child’s school, feel connected with community resources in addition to other

families within their community, and feel more confident in their parenting abilities.

F&STWORKS is initially to be run by F&ST staff and a “parent advisory council of graduates”

(McDonald, et al., 1997, p. 145). During F&STWORKS, families become more self-sufficient in

terms of scheduling meetings and determining the content and topics of the meetings which are

held in neighbourhood clusters. Families increasingly set the agenda of the F&STWORKS

meetings while the parent advisory council monitors the budget and meeting location

(McDonald, et al., 1997).

The eight-week portion of the F&ST program becomes a “bridge for moving new families into

the F&STWORKS program” and “the initial F&ST experience builds interdependency among

participants thus increasing the likelihood that they will join the F&STWORKS program”

(McDonald, et al., 1997, p. 145). The purpose of F&STWORKS is to create a “local association

of parents” so that parents who were not previously involved in their communities or within their

child’s school become more actively involved and feel able to “take more risks” because they

feel supported by others in their new social network (McDonald, et al., 1997, p. 145). As more

and more families graduate from F&ST and join the F&STWORKS portion of the program, the

local community network grows and strengthens. F&STWORKS then, builds families’ capacity

to become leaders within their own community (McDonald, et al., 1997).

Some Research on F&ST The majority of research done to evaluate F&ST and its outcomes has been conducted by the

creator of F&ST, Lynn McDonald. She is well published on the outcomes of the program

including long-term outcomes and has compiled findings across program sites. Also, in

partnership with other authors, she has researched the impacts of F&ST on various cultural

communities including Latino communities (2006), Aboriginal communities (2003) and on

teenaged single mothers (2008) in the United States. Findings suggest that the program is

beneficial for various cultural communities and the research has also allowed cultural

considerations for the program to be integrated. While there is also literature on several of the

9

sites in Canada that have implemented the F&ST program (Annual Program Report, 2006-2007;

The Family Centre Annual Report, 2007-2008; The Vanier Institute of the Family, 2006), it is

difficult to locate and there is not a lot available.

Based on the literature that is available, it is evident that the F&ST program has long-lasting

outcomes and indeed benefits the families that are involved in the program. For example, in

Manitoba, the F&ST program has been running since 1997 and program staff have reported the

outcomes of the program on families, including: increased family cohesion, reduced social

isolation, and increased connections to the school. Further, participants of the Manitoba F&ST

felt that they had become included in greater support systems and had increased involvement in

their child’s school (i.e. as volunteers, on parent council, etc.) (The Family Centre, 2007).

An annual report from Winnipeg (Families and Schools Together, 2007) outlined similar findings

and also dedicated an entire section to the developments, activities and outcomes of the

F&STWORKS portion of the program specifically. For example, in this F&STWORKS program,

the F&STWORKS (F&ST graduates) parents were supported in: developing a voluntary Parent

Advisory Council for F&ST consisting of parent partners from numerous communities offering

F&ST; developing a manual which includes “guidelines for organizing a meeting, managing the

financial affairs of a volunteer group and a list of possible activities for the monthly

F&STWORKS meetings” (p. 1, App. C); developing job descriptions and assigning responsibility

for planning and implementing F&STWORKS at various sites; having F&STWORKS

coordinators problem solve with issues and concerns at F&STWORKS get-togethers, and;

having F&STWORKS coordinators attend training in the Parents Assist Learning and Schooling

workshop series developed by the Canadian Home and School Foundation to learn how to

“Parent with Confidence”. In some instances, where city-wide events have taken place for

families in F&STWORKS, over 300 parents and children have attended.

An executive summary including the Canadian National Evaluation of Outcomes 1996-2004 put

together by Family Service Canada (2004) reported that data from 187 program cycles across

Canada showed the F&ST program provides immediate positive outcomes. These outcomes

include improved behaviour of children, improved emotionality of children, increased familial

cohesion, reduced social isolation of parents, and increased parental involvement in their child’s

school.

10

An article by McDonald and Sayger (1998) investigated the longer term impacts of the F&ST

program on single mothers, looking at outcomes at six months and two years after program

cessation. The authors found that the program indeed had long-term positive impacts and,

particularly for parents, the program provided social supports and empowered them to become

involved, in some instances, in leadership roles (i.e. Parent Advisory Council for F&ST). This

finding is important because “disenfranchised, poor, single mothers” were the program

participants involved (McDonald & Sayger, 1998, p. 83).

In 2009, several outcomes of children and families in an experimental study were investigated

(Kratochwill, McDonald, Levin, Scalia & Coover, 2009). Children with behavioural issues from

kindergarten to grade three were assigned to either the F&ST program or continuing with school

special education and assessed immediately and at one year following program cessation. The

findings indicated that the F&ST program participants faired better in terms of family

adaptability, and improved behaviour of children. Further, only one child involved in F&ST

required special education services in the future. The authors suggested that the program

results in some positive short-term outcomes for children.

The research on the F&ST program reveals that the outcomes are positive and both short- and

long-term outcomes are evident.

Past F&ST Evaluation Processes at Mosaic For the past several years, F&ST has used a pre-post non-experimental design with two raters

(parent and teacher). The standardized evaluation instruments have included: the Behavioural

and Emotional Rating Scale (BERS); the Family Adaptability and Cohesion Evaluation Scales

(FACES III); the Isolation Subscale of the Parenting Stress Inventory; the Witte Parent Survey,

and; the Families and Schools Together Program Evaluation by the Family, an open-ended

questionnaire providing the family an opportunity to explain how the program impacted their

lives. Parents and teachers completed the BERS, FACES III, and a survey on parent

involvement before the program began and after it ended. The measures were paired to

determine the level of change (Family Service Canada, n.d.). F&ST staff collected the data, and

in the past, a consultant analyzed and reported the data.

11

Purpose of the Current Evaluation With the help of the Centre of Excellence, Mosaic has had increased resources to implement a

more thorough and meaningful evaluation of the F&ST program. The challenges with the past

evaluation approach for the F&ST program were twofold: the approach was felt to be

inadequate in reflecting the nature of the program (collaborative and relationship-based), and

the findings were not reported in ways that could be easily understood by program staff

(intended users). Up until recently, a consultant would analyze and report the results in a short

report that included each school’s results. The results were shared in a PowerPoint slide during

F&ST training sessions, but little attention was generally paid to school reports. The quantitative

outcome data that was produced, while positive, was not found to be particularly useful to staff.

Recently, a teacher mentioned to a F&ST staff member that the surveys she completed did not

allow her to capture the stories of success she has witnessed in the program. The current

project sought to improve the approach to evaluation in the F&ST program and increase staff

and participant interest in being involved in evaluation activities. With the resources from the

Centre of Excellence, Mosaic re-engaged F&ST staff, partners and participants in implementing

evaluation and broadly sharing the results with staff, participants, partners, community, funders

and potential funders. Involving F&ST participants was an important part of connecting goals of

the program with evaluation.

Methodology Methods Used in the Current Evaluation The evaluation of F&ST for the Fall of 2009 incorporated several new methodologies not

originally included as part of the evaluation process including interviews and Photovoice, each

of which will be discussed in more detail.

A research design was developed in September 2009 and data collection took place from

October 2009 until May 2010. Both “methodological” and “investigator” triangulation were used

(Patton, 1980, p. 108): several methodologies were used to collect data and several

researchers were involved in collecting data (observation data, interview and Photovoice data)

respectively. A logic model for the F&ST program and an evaluation framework is included in

Appendix A. The evaluation methodology was designed to answer the research question “does

the F&ST program result in the changes it was designed to?” as well as a process evaluation

12

question, several evaluation questions directed at measuring short-term outcomes and several

evaluation questions directed at measuring long-term outcomes (see Appendix A).

.

Data was collected by several researchers (4) in the Research and Development Department at

Mosaic. Interview data was collected using purposeful sampling, the intent of which was to

gather a “breadth” (rather than a “depth”) of knowledge (Patton, 1980, p. 98) Researchers were

interested in gathering a breadth of knowledge to ensure that participants involved in the

evaluation represented all roles within the F&ST program (i.e. F&ST staff at Mosaic, F&ST staff

and volunteers, parent partners, Public Health personnel, teachers and other school personnel,

and past F&ST families) and as such, evaluation participants were purposefully selected from

each of roles (rather than hearing a lot of information from individuals from only a few program

roles). Another purpose for gathering a breadth of knowledge rather than a depth of knowledge

in this evaluation was to help determine what information should be gleaned by using a more in-

depth knowledge collection strategy in future evaluations of the program.

Researchers were responsible for analyzing and reporting interim and final data. It is important

to note that all participants that were invited to be interviewed accepted – none declined.

Participant Observation In the Fall of 2009, one researcher received the two-day training necessary to become a

certified F&ST volunteer which helped to develop an understanding of what the F&ST program

sessions included and the roles of all F&ST personnel. The two-day training assisted in this data

collection as increased knowledge was generated prior to observation regarding program

design, how it was to be implemented and the roles of the persons involved in the program.

Detailed information about how the program should be implemented as well as how effective

program implementation was taught in training was shared with other researchers and

afterward, several members of the research team observed the program running at several sites

throughout the Fall of 2009. The objective of this data collection strategy was to purposefully

sample each activity of the program to observe program participants and staff behavior. Greater

emphasis was placed, however, on “tracking” F&ST staff (Mosaic staff, program staff and

volunteers) with the purpose of following staff “around during their usual daily routines and

watching their activities and the other people they interact with” (Berg, 2007, p. 196) in order to

learn how closely the program matched in practice with training. With the research question

13

and evaluation questions in mind, program participants were observed to see if changes took

place over the course of the program (i.e. more interaction between family members, parents

providing instructions to children, etc.) and staff were observed to determine how closely the

program matched the intended model (i.e. staff whispering instructions to parents, etc.). In

addition, several researchers, with participant permission, took photos and video clips during the

program as a form of visual data. This was done while attending several of the actual program

sessions and photographing the activities engaged in by program participants and staff in order

to share this information through a different form of media (i.e. pictures and videos). This data

collection method allowed the researchers not only to collect photographic information on the

program, but also to observe numerous sessions of the F&ST program in addition to a

F&STWORKS session running in actual practice. Further, this data, in conjunction with the data

collected through interviews and the training offered for F&ST staff, gave a more accurate

understanding of what the program should look like in practice.

With regard to the F&ST program sessions, several researchers observed numerous F&ST

sessions in the Fall of 2009 in all four schools offering the program as well as one

F&STWORKS session. In addition, over 200 photographs and videos were collected from the

program sessions.

Data collected through observation were analyzed qualitatively to compare observational data

between researchers and findings were used to augment information gathered through interview

and Photovoice techniques.

Interviews Semi-structured interviews were conducted with a variety of persons involved in the program

including participants whose graduation from the F&ST program spanned several years, F&ST

staff and volunteers, school personnel, Public Health personnel, F&ST parent partners, and

F&ST staff employed at Mosaic. Thirty-three interviews (not including Photovoice participants)

were conducted by researchers in the Research and Development Department at Mosaic.

Interviews were conducted one-on-one via telephone or face-to-face and took approximately a

half hour to complete (see Appendix C and D for Informed Consent documents and Appendix B

for interview questions). Interview questions were reviewed prior to implementation by all

members of the Research and Development Department as well as several members of the

Mosaic F&ST staff.

14

With regard to participant recruitment, in some instances, such as with past F&ST families,

there were many potential families to invite to participate making it impossible to recruit them all

so both purposeful random sampling and convenience sampling were used for these

participants. Names and contact information was gathered for most families that had

participated across the years of the program and contact was attempted for many (purposeful

random sampling). Only the families who could still be reached through the contact information

provided were contacted for an interview (convenience sampling). The researchers wanted to

learn about the short-term impacts of the program as well as the impacts longitudinally which

would involve talking to recent graduates of the program as well as those who had graduated

over the years, but only the families that could still be reached using the contact information that

F&ST staff had were invited to participate. However, in some program capacities/roles, all

individuals (i.e. such as all F&ST staff employed at Mosaic) were interviewed because there

were fewer individuals in that role and it was possible to interview them all.

Data was analyzed using qualitative research techniques (coding) to search for and highlight

themes across evaluation participants. “Individual variation” was sought in how individuals

involved in the F&ST program, in various capacities/roles, felt about the program including its

successes and areas for improvement, and how well the program meets its intended goals

(Patton, 1980, p. 99). Interview participants in many roles (i.e. F&ST families, school personnel,

Public Health personnel, Mosaic and F&ST staff) spanned the numerous years that the program

has been operating and participants represented the various schools involved in the program

(represented several different sites).

Photovoice Data for the F&ST program was also collected using a technique called Photovoice. Photovoice

is a data collection technique that includes offering a camera to a number of participants and

asking them to capture a personal or community experience or issue through pictures (Berg,

2007; Wang & Burris, 1997). Photovoice is most often used to address community issues where

individuals relay messages about their community’s successes or failures on a particular issue

to persons such as government officials through images coupled with a story about the picture

(Berg, 2007; Wang & Burris, 1997). According to Wang and Burris (1997), Photvoice “entrusts

cameras to the hands of people to enable them to act as recorders, and potential catalysts for

15

change, in their own communities” (p. 369). To our knowledge, Photovoice has never been used

to evaluate a program making this a unique way to capture program feedback.

F&ST families participating in the program in the Fall of 2009 from Wilson Public School,

Howard Robertson Public School and Cederbrae Public School were invited to use Photovoice

to capture their experiences of the program. Sixteen families agreed to participate in

Photovoice: six from Wilson, six from Cederbrae and four from Howard Robertson. One

participant from Cederbrae ended up not participating due to irreversible damage done to the

camera provided. Eleven of the 15 families who were involved in Photovoice were interviewed –

it was not possible to connect with the remaining four families. All participating families were

given a digital camera and were asked to share their pictures and offer their stories of their

experiences in the program. F&ST families involved in the Photovoice data collection technique

were also asked the same interview questions as all other F&ST families interviewed during the

evaluation in addition to questions about their experience of participating in Photovoice as a

research methodology.

Photovoice participants were recruited by both Mosaic F&ST staff and researchers from Mosaic:

Mosaic F&ST staff first introduced the Photovoice method to the F&ST families at all three

schools and then introduced the researchers who offered more specific information. Interested

families were invited by the researchers to attend an information session about Photovoice that

would be held at a location near their respective schools (see Appendix F for a recruitment

flyer). Thirteen interested families attended two information sessions about Photovoice and

agreed to participate (see Appendix G for the Informed Consent form for Photovoice). Three

additional participants were recruited separately because they were unable to attend the

information session but were offered the same information as those who did attend (i.e.

informational handouts and informed consent document) (see Appendix H for information

slides).

After the program finished, families were invited to attend a focus group to share their pictures

and “accounts” of the photographs including why the pictures “were taken, what the image

means to the individual, and what they intended the photograph to depict” (Berg, 2007, p. 234).

This technique offers participants the opportunity to share information that they, and not the

researcher, have prioritized (Berg, 2007). Mosaic F&ST staff assisted researchers in arranging

16

the final focus groups and interviews with families (where they shared their pictures and stories)

(see Appendix I for the focus group recruitment flyer).

Six families attended two group gatherings for families involved in Photovoice in their respective

community (i.e. all families from Wilson school met at a location near the school) – families

shared their photographs and stories individually with a researcher from the Research and

Development Department and not in a group setting for purposes of confidentiality.

Approximately eight children were also involved in sharing their family’s pictures and stories and

commenting on their perspectives of the F&ST program (answering interview questions). One-

on-one interviews were conducted with the remaining five families (see Appendix J for the

Informed Consent document for Photovoice focus groups/interviews and Appendix B for

Photovoice and interview questions). Children were invited to attend the focus groups or

interviews. There was a plan to offer these children the option of a book to draw in to capture

their stories and experiences of F&ST while their families shared and discussed pictures.

However, these books were never used because all of the children in attendance at the focus

groups participated with their families in sharing pictures and stories.

Photovoice data was analyzed using participatory action research techniques. Families were

provided a camera to collect data by taking photographs of what they believed would indicate

successes and areas for improvement for the program. Families selected the photographs that

they wanted to share and discuss with researchers at Mosaic. Photovoice also allowed an entire

family to be involved in the data collection and analysis process – all family members could take

pictures of what they felt were important aspects of the program and share the pictures that they

wanted to. There were several instances in which a whole family was involved in data collection

as well as with sharing the pictures with researchers. The stories shared by families were

captured like an interview: semi-structured questions were asked including the same questions

as for other F&ST families, but additional questions were also posed that were directly related to

Photovoice to gather information on how families felt about this particular data collection

technique. This data was analyzed using qualitative analysis (coding) to search for common

themes across Photovoice participants, and compared to information provided across all other

F&ST families interviewed. According to Berg (2007) it is often the case that the Photovoice

technique is used in conjunction with other research techniques such as photographs taken by a

researcher, and also in this case, used in addition to interviews.

17

Surveys Some, although not all, of the standardized instruments originally used to evaluate the F&ST

program were also included in the current evaluation. Some of the survey instruments have not

been used for the last several years due to poor response rates from participants (i.e. a lack of

understanding about the meaning of the questions particularly with participants whose first

language was not English, difficulty with getting participants to complete the surveys, etc.). The

instruments included in the current evaluation were the Behavioural and Emotional Rating Scale

(BERS) filled out by teachers before and after the program through which the measures were to

be paired to determine the level of change; the Family Adaptability and Cohesion Evaluation

Scales (FACES III) (also pre-post test comparison) filled out by families, and; the Families and

Schools Together Program Evaluation by the Family which is an open-ended questionnaire

affording the family an opportunity to explain how the program impacted their lives. The

measures were to be paired to determine the level of change. F&ST staff collected the data to

be analyzed. Copies of the surveys have not been included in this report for copyright reasons.

Methodology Limitations The purpose of a triangulated data collection strategy including interviews, observational

techniques and cumulating all quantitative data was to answer the research question “does the

F&ST program result in the changes it was designed to?” according to the aforementioned

description of the program, program goals, and intended outcomes. It was also used to make

the data collection and findings more interesting, meaningful and useful for all participants

involved in the evaluation, particularly F&ST families, F&ST staff and volunteers and F&ST staff

employed at Mosaic.

One limitation of the current evaluation methodology would be the purposeful sampling

technique used. Specifically, it was of interest to gather a breadth of knowledge (to hear from

many individuals with various perspectives of the program - from staff to participants to others in

the community) but due to the large number of roles that the program incorporates, there were

only a few individuals represented in each of the capacities/roles. For example, only three

Public Health staff members were interviewed. If, for example, more individuals were

interviewed but fewer program roles were represented, more information would be known for

the roles that were targeted. In contrast, however, the wealth of knowledge gained from

interviewing individuals from all program roles provided information not previously known and

provided a greater insight into the program – its activities and its outcomes. Indeed, according to

18

Patton (1980), breadth and depth are both important and neither is better, it is simply a choice of

which fits better and in this evaluation, seeking a breadth of information was considered more

important for reasons previously discussed.

Another limitation was the use of convenience samples to connect with past program

participants - participants that could easily be connected with were contacted for an interview.

This meant that more participants were interviewed that had recently graduated the program

than those that had graduated several years ago and thus, more short-term findings were

investigated than longer-term findings. This limitation, however, points to a key finding, which

will be discussed in greater detail to follow, that program participants who had graduated

several years ago may have been easier to contact had they still been involved in

F&STWORKS, the continuation of the program after the end of the initial F&ST programming.

Results

The findings outlined here are the major themes that have been discovered in the data that

were noticed across all participants involved in the evaluation (i.e. from all roles in the program).

The findings were analyzed using qualitative methods and a triangulation of various data

collection techniques (interviews, Photovoice, observation). These findings are organized into

the strengths of the program (successes of the program and areas of the program that are

beneficial to families) and challenges of the program (areas for improvement and aspects of the

program where short- and/or long-term outcomes are not evident).

Strengths of the Program These are the areas of the program that were found to be beneficial to families in the program,

demonstrated an outcome intended by the program, and were found to be successful through

interview and Photovoice data.

Family Time Many participants discussed the benefits of having an opportunity for families to come together

for an evening. One of the positive aspects of the program is to help families learn how to better

connect, communicate, and support each other as well as provide an opportunity for them to

spend time as a family.

19

“for some of these families who might be single parent families or experiencing some

frustration with their children at home, it’s a night of fun where they can kind of forget

about their problems”

Meeting New People and Building Community

The program allowed people to familiarize themselves with their community and meet new

people in their neighbourhoods including new Canadians. The program also provides an

opportunity for families to connect with others: children could meet other children and parents in

particular could discuss parenting concerns with other parents and offer each other support in

an inclusive manner.

“a lot of parents who are new to the area, or who are new to the school, get to know

other families…it creates a support network in our area”

Connection to the School

The program increased the level of connection between parents, their children and the school.

The connection to the school and to school personnel allows children and parents to become

familiar with the school and school personnel and feel more comfortable in the school system.

“I’ve learned the value of making connections with the families that are coming new to

your school and hooking them in as early as possible to maintain their involvement in the

school”

Involvement in the Program F&ST staff and volunteers stated that they enjoy giving parents the opportunity to have a fun,

family night, being able to support them in modeling good communications and parenting to

their children and enjoy developing relationships with the families involved in the program.

Teachers also indicated that they enjoyed this as well as getting to know children outside of the

classroom setting.

Inclusivity Several participants mentioned that the program offered a chance for all individuals, including

those with financial difficulties, and single parents, to participate. Further, the program provided

space for newcomers to Canada to practice and improve their English skills.

Supporting Enhanced Parenting Skills

20

The program offered support to parents by helping them enhance their parenting skills and

assisting them in learning how to manage their child’s behaviour.

“Parents are seeking permission to tell their children what to do and to guide them -

they’re thinking that that isn’t their job, and they see that they are the one telling their

child what to do through this program then it really kind of turns things around in a

positive way – the parents have more confidence with telling their kids what they need to

do and have that that leadership role in their families which is something that some of

them lack”

Growth in the Program Changes were observed by school personnel, F&ST staff and volunteers and Public Health

personnel in families and individual family members over the course of the program. In

particular, increased communication was seen amongst family members and toward others

such as teachers and F&ST staff.

“the children or the parents who come at the beginning who don’t talk to you, barely talk

to you, don’t make eye contact, are guarded through the activities and by the end of the

eight weeks, they call you by your first name, they come and hug you when they get

there…you can sense the enjoyment”

Counselling Services Several participants discussed the importance of the program for providing support much like

the type of support that might be offered through group counselling.

“ we are not in the business of rescuing children from their parents...we are in the

business of strengthening parents. And I always thought that this program should have

been a model for all our children’s group that we do at the agency” Respondents across all components/roles of the program stated that the program was positive

and helpful to families involved. The program is also positive for those involved in administering

it such as for F&ST staff members and volunteers, school personnel and Public Health

personnel for a multitude of reasons.

Challenges of the Program While it is evident that the program offers numerous benefits for families involved, respondents

did speak to the areas that could be improved. These findings also indicate that short- and/or

long-term outcomes are not evident in all aspects of the program.

21

F&STWORKS

Overall, among F&ST families, there was a lack of knowledge of, or engagement in, the

F&STWORKS part of the program, originally intended by McDonald et al. (1997) to be the main

focus of the F&ST program in building community engagement, strengthening family systems

and thus decreasing problematic behaviour in children. Further, very few interviewees and

Photovoice participants discussed F&STWORKS suggesting that this is not a core component

of the program. Indeed, several interviewees stated that there is little turnout for or little

knowledge of F&STWORKS which indicates a lack of focus on this aspect of the program.

“F&STWORKS is…where we need to focus our attention. We don’t get great turn out.

The best turn out last month where we had 6 families come. That was huge for us,

That’s full.”

“if I were to say something that could be improved upon, how they do

F&STWORKS…they don’t do things that are interesting”

“I’ve never heard of that [F&STWORKS]”

F&ST staff are trained on F&STWORKS (i.e. the purpose of F&STWORKS, the theory behind it,

etc.) but in practice, this portion of the program is not very strongly promoted and very few

families become involved which may be one reason why some long-term outcomes are not

evident.

Time Commitment Many participants mentioned that the time commitment that is required, particularly of F&ST

staff, volunteers and school personnel was challenging. Many found it challenging to commit to

being available for the full eight or ten weeks of the program.

“the time commitment may involve in teacher involvement…ten weeks, even eight weeks

[is long]”

In contrast, however, many of the families who had participated in F&ST stated that the program

was too short. For example, numerous families felt that at the completion of the program, they

were not prepared for it to end.

“I think if it was a little bit longer, we would have probably established much deeper

connections right? With different families”

These findings indicate that there may be insufficient emphasis being placed on F&STWORKS.

Involvement in F&STWORKS allows families to continue to learn and practice many aspects of

22

the program and provides ongoing networks of support following the end of the initial 8-week

F&ST sessions. However, with an increased emphasis on this second stage of the program,

F&ST and Mosaic staff, volunteers and teachers would actually be required to offer more of their

time to the program (approximately two years) which could be seen as positive or negative in

terms of time commitment/ could increase their concerns regarding the challenges involved in

the time commitment required.

Program Structure One very important aspect of the program is to provide support to parents so that they can

enhance their parenting skills. This is done through tangible supports offered but also through

F&ST staff, volunteers, teachers and other F&ST team members relinquishing their typical

“roles” and providing instructions throughout each session (i.e. activity changes, behavioural

feedback, etc.) quietly to the parents so that all instructions given to children come directly from

their parent (McDonald, et al., 1997). F&ST team members are to whisper instructions to

parents rather than offer them directly to children (Family Service Canada, n.d.). Feedback from

F&ST team members indicated that it is difficult for many to forget their normal roles, particularly

F&ST staff and teachers, and to engage in the required activities of the program. Some

interviewees suggested that because of this challenge, the structure of the program had in

some cases been compromised. For example, if staff, volunteers, teachers or other persons

involved could not detach themselves from their typical roles (whether their role is being a

teacher, a counselor, etc.) and assist parents in actual parenting, the standards of the program

as outlined in the F&ST Program Workbook (Family Service Canada, n.d.) were not being

followed and the goals of the program not sufficiently being met (i.e. to help parents learn and

practice parenting skills).

“I think it’s something people forget and it’s hard for professionals like teachers to get

their head around…whispering in a parent’s ear to get them to do it [parent]…the intent

is to…empower the parent to let them know ‘we have faith in your parenting

capacity’…[this is] something that does get dropped”

Several interviewees also described the importance of remaining true to the program structure

and how the format is necessary to achieve the intended outcomes of the program.

“this is where I feel it would be important to keep in contact with Lynn McDonald and,

you know, her original design was so based on choosing activities that were fun filled,

strength based, results oriented...she had a set of principles that were in many ways

23

common sense principles – but she was able to take her principles and to see what has

worked in practice”

It was also suggested that the role of the team leader should include more structured

observation to ensure that activities are being followed.

“to strengthen the role of the team leader…to mentor and observe…how we’re doing as

team members”

Participant Recruitment Several challenges were discussed with regard to recruiting participants. A particular challenge

mentioned by several teachers was recruiting participants into the Fall portion of the program.

This was particularly difficult both because parents were not familiar with teachers, making trust

and program buy-in challenging, and because teachers were not familiar with children, making

referrals based on behaviour challenging. “it’s a struggle to get parents because our program is right at the beginning [of the

school year]”

Several school personnel, F&ST staff and Mosaic F&ST staff discussed ‘repeat’ F&ST families

and the difficulty in negotiating how often to let participants return to the program versus recruit

new families. Some families were allowed to repeat the program because they did not fully

understand what was being taught. “some of them [families] need to be here more than once to really grasp what is being

taught”

“some families are repeats which is nice…just because I think it gets talked up in the

family…I think our parents enjoy that aspect of it”

Having families repeat the program may be problematic because it prevents new families from

joining. Addressing this issue may be one solution to the suggestion that the program should be

offered to more people.

Program Outcomes Many interviewees discussed the shorter-term impacts of the program including meeting new

people (for children and parents), getting to know the school, and being involved in activities as

a family. It was suggested by individuals from different program roles, however, that the

program may not have led to much change, especially over time.

24

“I feel like I’ve never been like, “wow, I really see that all of that time and effort has made

a difference for that kid”

“Has it [the program] improved my life? Has it improved our family? I mean it did at the

time. Has there been like a long lasting improvement? No. I don’t think it’s made any

significant long lasting impact. It was something that was positive at the time”

Families who had been through the program also failed to demonstrate lasting changes. Also,

parent-child interactions were not greatly impacted because improvement noted may in fact be

due to families becoming more familiar with the program activities themselves. “as families got better at doing the program so that they got better doing charades, or

um, you know whatever. I think they, started going through the motions. Sometimes it

was just uh, there wasn’t a lot of thought put through, and they could just kind of go

through the motions to get to the next point. I think we needed to be more diligent about

encouraging mom and dad to set the tone to take the time to play the charade or to do

some drawing”

The idea that families may only be improving as a result of becoming familiar with activities

coupled with the finding that there is little improvement over time suggests that the program may

not be achieving all short- or long-term goals. It may also mean that children and families

recruited to the program might be functioning fairly well as a family prior to entering the program

and are not in need of much support.

Purpose of the Program All interviewees were asked to describe how they perceived F&ST and the purpose of the

program. Most (from all roles) described it as an opportunity for families to spend a night

together and join each other for a free meal. Others included that it was an opportunity for

families to get to know others and their schools. Some also described it as an opportunity to

build children’s skills.

“I don’t make it [family time] a formal thing or organized at a particular point in the

evening [since the program]…I guess we’ve kind of semi-kept up with it…I like the idea

of having to go out”

These findings do not, however, incorporate the longer term impacts of the program, focused on

during F&STWORKS which are improving the behaviour and success of children through

strengthening families, building greater social networks, getting to know resources, and

strengthening the community (McDonald, et al., 1997). This, along with the findings that suggest

25

that many respondents have little engagement in, or knowledge of F&STWORKS, strongly

indicates that emphasis is placed on the eight-week portion of the program and that as such,

many long-term outcomes are not achieved.

Staying Connected to New People

While the program allows children and parents to meet new people within the program, previous

F&ST families stated that they did not spend much time with other parents once the program

ended. “we don’t call or anything but when you see them around the school yard and stuff you’ll

stop and chat”

This finding indicates that while parents and children do have an opportunity to connect with

others and some do remain connected, many do not develop long lasting relationships - one of

the goals of the F&STWORKS portion of the program (McDonald, et al., 1997). Indeed, one of

goals of F&STWORKS is to empower numerous parents within a given community to become

leaders in that community and work as a collective to improve their neighbourhood (Family

Service Canada, n.d.; McDonald, et al., 1997).

Observational Findings The observational findings corroborate several of the suggestions made by participants.

F&ST Sessions

It was observed at F&ST sessions by several researchers that several of the intended goals of

the program were being achieved during the program. For example, families spent time together

eating a meal and participating in various program activities and appeared to enjoy their

evenings at the program. As the program progressed, it was observed that parents began to

engage with other parents and children spent time playing with other children.

It was also observed, however, that the structure of the program was not closely adhered to.

There appeared to be gaps and inconsistencies in parental support during the session. For

example, transitions in the program (i.e. activity changes) were offered by staff rather than

whispered to parents who would then offer them to their children. Further, behaviours

considered inappropriate (i.e. children running around freely and not engaging in the program

activities, etc.) were not often reprimanded and when they were, were corrected by staff rather

than by parents with staff assistance. These findings were mirrored in the information offered

through interview data. In addition, a lot of “standing around” was noticed at all program sites by

26

F&ST team members suggesting that there may be alternative ways to use human resources

that would better achieve program goals or that there may have been too many human

resources at each program site. These observations were similar at the one F&STWORKS

session that was observed.

Previous Evaluation of the Program

Evaluation was both criticized and praised by interviewees in all program roles. Several

interviewees stated that they understood the importance of having an evaluation, but many

specifically disliked the original pre- and post-test format of evaluation.

“the paperwork…I mean more specifically… all the questionnaires, and as someone

with a research background, I totally totally understand the importance of evaluative

programs to see if there’s a change from point A to point B but I think the problem lies in

the fact that the…the questionnaires are quite long and I think there’s three or four of

them and it’s quite onerous on families and but also even very literate people don’t like

filling out forms, it’s much more difficult for families who are new to Canada…it’s difficult

to get families to do it”

Filling out questionnaires was found to be difficult, particularly for teachers because they found it

challenging to determine how to rate the children based on the questions available in the

survey. Particularly when the F&ST program ran in the Fall, teachers were still unfamiliar with

children and therefore unfamiliar with their behaviour. It was felt by most respondents, however,

that evaluation was important in order to glean insight into what aspects of the program worked

well and what improvements could be made.

New Evaluation Process

It was felt by all researchers based on direct feedback and the amount of information gleaned

about the program, that incorporating interviewing and Photovoice techniques in particular as

well as observational data collection strategies into the evaluation structure was beneficial. New

and more accurate data was gathered through more participatory evaluation models. For

example, only by interviewing participants could it be learned that teachers struggled with filling

out pre- and post questionnaires about students that they hardly knew.

Participants who were involved in Photovoice found this way of collecting information to be fun

for them and their families. It promoted buy-in to the evaluation process and captured stories

27

that may not have been captured otherwise. It also gave participants a chance to take pictures

of their families which they not have taken otherwise.

“I give it to [a friend] to take a picture from us in my own camera because I would like for

memorizing…for memory for also when my kids growing, I will tell them what we have

done and where we go and I will show them the picture that I took”

In fact, some Photovoice participants wished that the camera would have been given out earlier

in the program to capture more stories.

“it was a good idea but I think, I just hope you would get the camera earlier…you miss

opportunities”

Photovoice was also felt by the researchers involved in the project to be a good way to collect

data from participants. Having pictures shared by families worked to develop rapport quickly and

revealed more information about the lives of participants. Further, participants were in control of

what pictures they wanted to take and share and thus empowered families to tell their stories.

Other Information As mentioned, the purpose of this evaluation was to answer the research question “does the

F&ST program result in the changes it was designed to?” as well as several evaluation

questions. As can be seen based on the findings, while families do enjoy the program (i.e. find it

a fun evening with their family) and often feel better connected to the school and their

community, it is not clear that the program does result in many of the short- or long-term

changes that it was designed to for a variety of reasons including program infidelity. For

example, families indicated that they enjoy the program and this answers the process question

about the level of satisfaction of program participants. There is not clear evidence that the

program results in increased support for families, increased connectedness to the school or

community, increased ability to handle life’s challenges, increased ability for parents to act as

primary prevention agents for their children nor that it improves success in school for children or

improves their behavior in school. While some of these outcomes are evident immediately

following the program (i.e. increased connectedness to the school), these do not always remain

after a long period of time.

It is important to note that one data collection technique originally included in the data collection

strategy, an accumulation and analysis of quantitative collected over the course of the F&ST

program, was not executed. This was because the collection of data that took place during the

current evaluation suggested that the data collected in the years prior may not be a good

28

indication of the successes or areas of improvement for the program. For example, part of the

prior evaluation included collecting data from teachers but it was discovered through the current

evaluation that in many instances, teachers do not know their students well and thus, do not

have an accurate understanding of how their behavior would have changed over the course of

the F&ST program in order to judge program success or failure. As such, consolidating data that

may not be entirely accurate did not seem desirable. Instead, more interviews were conducted

and analyzed. While the purpose of the data analysis was to seek “individual variation” amongst

evaluation participants with respect to the roles in the program (i.e. variation among teachers,

among families, etc.), it was noticed that there were similarities in what was said across groups

in that several themes were heard across several types of program roles, across several

program years and across several program sites. For example, many participants, regardless of

their role in the program, what year they were involved and what site they were at, felt that the

program was simply a fun night out for the family which suggests a misconception about the

purpose of the program. Indeed, according to Patton (1980), this type of sampling (i.e. selecting

3-4 individuals that represent each program role) “increases confidence in common patterns

that cut across different programs” (p. 105) or in this instance, program roles, sites, and years

offered. Overall, more information was gathered this year about the program than any other

year that is likely more accurate and presents a greater picture of the program.

Discussion & Lessons Learned Sharing Information At the beginning of the evaluation, an evaluation advisory committee was formed. This

committee included Mosaic F&ST staff, members of the Research and Development

Department at Mosaic, and other community stakeholders involved in the operation of the F&ST

program. This committee was involved in assisting the researchers develop an evaluation

strategy and provided input and feedback to the researchers regarding the anticipated strengths

and challenges of the new evaluation strategy, any amendments necessary, and provided

feedback on the research tools (i.e. thoughts and feedback on Photovoice, the interview

questions, which questionnaires to administer, etc.). In addition, Mosaic F&ST staff were asked

to provide ongoing input into the research design and research tools and were actively involved

in participant recruitment for the evaluation. For example, Mosaic F&ST staff suggested that

since rapport had already been built between themselves and F&ST families, that they should

introduce to families Photovoice, the opportunity to be involved in Photovoice, assist

29

researchers in providing families with more specific details about the technique as well as help

to arrange the final interviews with families (where they shared their pictures and stories).

Learning One learning that occurred during the current evaluation was regarding the data collection

technique Photovoice. Specifically, researchers and Mosaic F&ST staff members discussed and

agreed on the difficulty they experienced in ensuring that the goal of the technique was

achieved: to take photographs of the experience of the program including areas of success and

areas for improvement. For example, Photovoice participants were reminded several times

throughout the course of the program by researchers and Mosaic F&ST staff to take images that

met the goals of this data collection technique but all families but one took pictures of more

things outside of the program than of the program itself (i.e. photographs that had nothing to do

with the program). While the purpose of Photovoice as a participatory technique is to allow its

participants to control the data they collect through photographs, it was evident that too much

freedom caused participants to veer away from the goal of the technique. Ultimately, each

family did have pictures that fulfilled the purpose of this technique but many more of the

photographs did not. This learning, however, was felt by researchers and Mosaic F&ST staff to

be very important to how the data collection technique would be promoted and monitored in the

future for program evaluation. For instance, a disposable rather than a digital camera would be

offered to Photovoice participants to limit the number of photographs a family could take in the

hope that most pictures would be related to the program. In addition, greater emphasis would be

placed on the purpose of Photovoice and on capturing what worked well about the program and

what could be amended rather than the “experience” of the program. It is important to note,

however, that overall, Photovoice was felt by the participants and researchers to be an

innovative and positive way to collect data as well as involve individuals in data collection and

with a few amendments, it would be very successful as a program evaluation technique.

The Evaluation Experience Prior to the commencement of the evaluation, there was a consensus between the Research

and Development Department and the agency’s Executive Director that research and evaluation

were of value and importance to the agency for learning about successes and areas for

improvement for all practices in the agency. Indeed, research and evaluation had become an

important component of all applications developed and submitted for funding by the agency.

There was also an assumption that agency staff fully understood and embraced this integration

30

of research into the practice of the agency. This evaluation project provided an opportunity for a

reassessment of the degree to which a research culture had been sufficiently embedded at

Mosaic and the steps required to further develop staff understanding and comfort with

integrating research knowledge into their work. While there is agreement regarding the value of

evaluation, it became clear to the Research and Development Department that there are

differing perspectives amongst staff of where, when and how much research and evaluation are

needed to support good practice. This evaluation prompted an in-depth discussion of research

and evaluation within the F&ST program and across departments, and in particular, the ways in

which the evaluation findings should be assessed and integrated in program delivery. In

addition, this evaluation prompted dialogue amongst staff about research validity and rigor –

staff sought clarification about what is involved in rigorous research and evaluation and so

demonstrated their curiosity and confusion about the process. This dialogue was productive and

will require an ongoing commitment to mutual learning as there remain various, often competing

opinions amongst agency staff about what constitutes rigorous research. Through this

evaluation process, therefore, it became apparent that a more clear and in-depth understanding

of what constitutes research in general, as well as rigorous research, is required for agency

staff. Indeed, if agency staff are unfamiliar with what research is, and what is involved in

conducting research, this leaves room for uneasiness in accepting presented findings and a lack

of clarity in how to learn from them.

This evaluation resulted in a debate amongst various staff members with regard to the

components of the program and the intended outcomes of the F&ST model. Specifically, the

evaluation provided an opportunity for staff to re-visit and re-consider program components

including the purpose of each component and whether each was achieving its intended

outcome in addition to whether the sum of all components were helping families achieve short

and longer-term outcomes. This evaluation project led to the mobilization of program staff

around the value of the program and each of its components but also provided an opportunity

for each staff member to reflect on how the program, as it is offered in actual practice, deviates

from the original design. Staff members were provided the chance to consider how significant

the deviation is and whether and how the deviation benefits program participants. The

mobilization of staff around the value of the program and its components, however, created

some challenges for researchers involved in the project to convey both program successes and

challenges equally. Indeed, given the passionate support and confidence in the program’s

effectiveness that many staff feel, there was some reluctance on the part of some to accept

31

findings that pointed to areas for improvement within the program. This had led to a degree of

mistrust in the evaluation findings on the part of some program staff that may require additional

dialogue to ensure that this does not create a barrier for making the required improvements to

the program to ensure it achieves its intended short- and long-term goals.

This evaluation highlights the strong need for individuals in leadership roles, including agency

Executive Directors and others in a leadership capacity, to understand and believe in the value

of research to build a capacity for, and a culture of, evaluation within their agency. Further, it is

important for those in leadership roles to have some understanding of what is involved in

conducting rigorous research and evaluation in order to continue to promote its importance and

relevance to staff. Indeed, it is not enough to assume agency staff appreciate or understand

research – frequent and consistent messaging and organizational learning about research and

evaluation would be required to increase interest, and ease distrust in evaluation.

This evaluation provided evidence that there must, within an agency, be a partnership between

agency leaders, staff and researchers to ensure an effective translation of findings into practice.

All agency staff should have an understanding of research, and more importantly, an

appreciation for research and evaluation in order to welcome all findings (successes and

challenges). Moreover, it is not enough to assume there is a culture of evaluation and that

agency staff understand research and evaluation (what research is, what it means to conduct

research, and how to use findings). Further, change can only occur when agency leaders and

staff are ready for, and interested in it. What became evident through this project is that in order

for this evaluation to lead to change, there must first be a culture of evaluation, or at least an

appreciation for evaluation in place within an agency.

The experience of working with the Centre was very positive. Researchers at Mosaic felt

supported throughout the process so that at any time, questions could be asked and answers

were immediately provided. Researchers also felt that information was constantly available

through the Centre’s website, trainings and through staff. Regular conversations with Centre

staff allowed for the researchers to keep connected with the Centre and provide updates about

progress made, ideas about the process of the evaluation and initial findings. The Centre also

allowed space for researchers to voice concerns about the evaluation and discuss any changes

made during the process. The most beneficial aspect of the Centre’s approach to planning for

an evaluation and reporting on the process, progress made and findings was the constant

32

encouragement to discuss the aspects of the evaluation that were challenging, required

amendment or that did not go as planned. This was particularly helpful because it acknowledges

that even when evaluation is planned in detail it does not always go as expected. The Centre

perceives that this is not a failure, but rather a chance for the agency itself and others to learn

about what could be done to remedy the problem next time. The Centre also allowed for many

agency staff to be involved in the evaluation process by creating space for all agency staff to

participate in online training as well as in the onsite visit at the beginning of the project. This was

beneficial because it offered staff an opportunity to learn about evaluation and research and be

involved in the evaluation process from the outset.

Conclusion & Recommendations/ Next Steps Recommendations for Others The initial recommendation flowing from the current experience of evaluation would be to

sufficiently assess the capacity of the agency to engage in and understand research and

evaluation as well as respond to and integrate findings prior to commencing an evaluative

project. This would include determining the actual level of interest amongst staff members in

evaluation, the level of comfort staff have with conducting an evaluation and how receptive staff

members would be to both positive and negative findings as a result of evaluation. For example,

how open would staff be to amending a particular part of their program or agency if it was

discovered that improvements were required? Further, it is important to determine the level of

knowledge that staff have about research and evaluation and provide ongoing opportunities for

learning so that with greater knowledge, fear or distrust are eased, and interest and buy-in are

increased.

An additional recommendation would be to design and implement various types of evaluative

strategies that may be unique and innovative. For example, in the current evaluation,

Photovoice was used as a data collection technique and while the success of this technique

was not entirely clear beforehand, it was determined that this is a very successful technique

according to F&ST program participants that could be used again to further evaluate this

program, or other programs at Mosaic. Using different techniques allows researchers and

evaluators to discover new, innovative and effective ways to evaluate programs, learn new and

valuable information, and gather greater knowledge and understanding of program participants.

33

Recommendations for the Organization A suggestion that could be offered, particularly to F&ST staff and Mosaic F&ST staff, is to

recognize that while participants involved in the evaluation offered information on what aspects

of the program need to be amended, agency staff are in a unique position to make these

amendments. Many people, including participants from all roles of the program, spoke about

how appreciative they were of the kindness, dedication and positivity of the staff and would

therefore likely be open to any amendments staff made to the program. The evaluation process

simply pointed to aspects of the program in need of improvement and while it is ultimately the

responsibility of the agency to ensure that these amendments are made, F&ST staff and Mosaic

F&ST staff are in a position to promote buy-in for change and growth within the program among

program participants (families) and program partners (i.e. school personnel, Public Health

personnel, etc.).

Another recommendation would be to consider the findings from the evaluation to determine

one of three options, each of which will be discussed in greater detail to follow: 1) amend the

practices in the program so that they follow the original program design and thus remedy issues

of fidelity, 2) re-vision/re-structure the program such that only the components of the program

with demonstrated success remain and re-name the program, or 3) cease offering the program.

Remedy Fidelity Issues

As discussed above, based on observational data, previous knowledge of the program acquired

through the standardized F&ST training and feedback from interview participants, the program

is not being implemented entirely as outlined in the F&ST Workbook, by McDonald et al. (1997)

or by other agencies running the program (Family Service Canada, 2007; Kratochwill et al.,

2009; McDonald et al., 1997; McDonald & Sayger, 1998; The Family Centre, 2008; The Vanier

Institute of the Family, 2006). It is important, according to Melde, Esbensen and Tusinski (2006),

to provide consider information pertaining to which program components contributed to

“success” and “failure” (p. 717). It is also crucial to include a full discussion of what proper

implementation constitutes in order to achieve the intended outcomes of a program. This is also

important to successfully replicate a program elsewhere (Melde, et al. 2006). Indeed, while

McDonald (1997) and the F&ST Program Workbook (Family Service Canada, n.d.) clearly state

that modifications to the program are acceptable to fit with the needs of the community, these

changes should not fundamentally alter the nature of the program and therefore, the core

components (i.e. activities, program duration and staff roles) should remain constant. It can be

34

argued that to date, the program has been altered too greatly to achieve the desired outcomes

and to be compared with other F&ST programs in the country. For example, the F&STWORKS

component of the program deviates more than any other area of the program from the original

design and as such, offers the fewest outcomes for participants. This is evident in the lack of

information that interviewees and Photovoice participants offer about either the successes or

challenges of this program component. It also explains the comments suggesting that this

particular aspect of the program requires amending made by few interviewees. If the core

components of F&STWORKS more closely matched the original design, the outcomes could

then be compared to other programs offered across the country.

Program Components to Keep for a New Program The findings suggest that there are numerous aspects of F&ST that respondents enjoyed and

felt were beneficial for families in the program. One option, then, is to re-structure the program

in order to keep offering the successful components of the program while removing the

remaining components and re-name the program. The following are the successful components

of the program:

Family Time - Allowing families to come together as a unit Meeting New People - Creating opportunities for people to connect with others Connection to the School - Increasing the level of connection between parents, their

children and the school Involvement in the Program - Providing an opportunity to staff and volunteers to be

role models for appropriate parenting and communication for children

Building Community - Families who are new to an area or to the country can

familiarize themselves with their neighbourhood and meet new people Inclusivity - All individuals, particularly those who living on low-income, are able to

participate and be offered support Parenting Support – There is potential to provide support to parents and assists them

in learning how to manage their child’s behaviour and increase the confidence of parents to

“parent” their children Counselling Services - Demonstrating the importance of counselling and offering

support much like counselling would but in a fun manner and within the client’s community

Cease Offering the Program

35

It was evident through this evaluation that there were numerous successes and positive aspects

of the program. It can be argued, however, that it was not clear that many short- or long-term

goals were being achieved and it may be appropriate to reconsider whether this particular

program is the most effective way to achieve the desired outcomes of the program.

Continue to Implement the New Evaluation Strategy Another recommendation would be with regard to the new evaluation strategy created to

evaluate the program and to utilize this, or a similar strategy in the future. A great quantity of

information was generated from including interviews, observational strategies and Photovoice

into the program evaluation. Interviewees from several program roles (i.e. school personnel,

parents, F&ST staff, etc.) discussed the difficulties they experienced in filling out pre- and post-

questionnaires but felt that evaluation is important. Photovoice in particular was well liked by

participants and researchers felt that interviews, Photovoice and observational strategies

provided information and stories about F&ST that would not have otherwise been captured.

Strengthen Evaluation Capacity The final recommendation would be to strengthen the capacity for evaluation within the agency

before evaluating any other programs. Further, a recommendation for individuals in leadership

positions, specifically, would be to persistently work to ease distrust of research amongst staff in

the future. This is important so that future suggestions based on evaluation project findings are

more positively received by staff members, that staff members will actively seek to be involved

in research processes and that both positive findings and suggestions for improvement are

perceived as equally important and desired among staff. Further, greater capacity for evaluation

and the perceived importance of research will, in time, result in staff members actively pursuing

areas of the agency that could be evaluated in the hope of future improvement.

The Centre The Centre provided more than sufficient access to assistance and education regarding

evaluation. Training was easily accessible both in terms of seeking information (i.e. on the

internet, etc.) but also in terms of having it provided in a way that could be easily understood by

those less familiar with evaluation. Further, Centre staff were always accessible if further

assistance was required. The greatest success of the Centre was to offer the opportunity and

support, specifically in this project, for the research methodology to include a data collection

technique not often used to determine how successful it would be for gathering information.

36

Knowledge Exchange Knowledge sharing activities originally included providing information to agency staff, volunteers

and partners as well as F&ST staff, volunteers, partners and program participants verbally (i.e.

staff meetings, face-to-face conversations, large scale presentation) and/or through writing such

as reports and the agency website. Knowledge exchange was also to take place with the F&ST

head office in order to share research findings.

Information about the evaluation process and findings were shared in a variety of ways

throughout the course of the project. An interim report of findings was developed in April of 2010

to share interim findings with the Executive Director of Mosaic as well as Mosaic F&ST staff.

The interim report included interview and Photovoice data collected as well as observational

data collected by April. It also included, based on interim findings, successes of the program as

well as recommendations for program amendments so that services for the children involved in

the program and their families could be improved and both short- and long-term goals could be

more effectively achieved. The Executive Director and Mosaic F&ST staff also had an

opportunity to discuss with the researchers their thoughts and feedback about the interim report,

findings and recommendations.

In addition, a large-scale event was held for Photovoice participants and their families in June of

2010. Specifically, the images that F&ST families that participated in Photovoice selected (one

image for each family) were enlarged and framed along with the “story” (a quote) that they

offered to describe the importance of the photograph and how it related to their experience of

the F&ST program. The event was offered on a Friday evening and Photovoice participants as

well as other F&ST families, Mosaic and F&ST staff, F&ST school personnel and several other

community members were invited to come and view the images. Information about Photovoice

as a data collection technique was offered to those in attendance and a slideshow of images

taken at the F&ST program by the researchers and Mosaic F&ST staff played during the event.

In addition, the display was made available in the front lobby of Mosaic for several weeks

afterward to offer agency clients and partners an opportunity to learn about the F&ST program,

Photovoice as a data collection strategy and of the research and evaluation being conducted by

the Research and Development Department at Mosaic. Children and their families involved in

Photovoice attended the event and were offered an opportunity to review and showcase the

picture selected by their family. This event, then, was an opportunity to empower children and

families by sharing broadly their experiences in the program and their ability as a family to take

37

creative, powerful images. Families were also offered their framed photographs and stories to

keep as a token of appreciation for their involvement.

Several researchers also attended an event entitled “Central West Regional Conference 2009-

2010: Meeting changing needs with emerging knowledge” in March 2010 at CHEO to share

their experiences of and some of the information gathered through the Photovoice technique.

The researchers developed a poster as well as a one page handout with information about

Photovoice, initial findings and contact information (see Appendix K for the one page summary).

This knowledge sharing event was of particular importance because it offered researchers an

opportunity to share with others the effectiveness of using Photovoice as a participatory action

research technique to engage participants, and particularly children, in collecting and reviewing

data. Photovoice as a research technique in this instance not only allowed children to be

involved in the process of collecting data (via taking photographs) but also allowed them to

share their experiences of a particular program with their own families and with researchers.

The process created opportunities for children and their families to connect with one another to

negotiate what photographs to take and in some cases, to discuss how these images reflected

what they, as a family, believed to be important information to share with others about the

program. The process also allowed several families to come together as a unit to select one

picture to be shared with others through the large scale event in June. Photovoice is a unique

way to capture the experiences, knowledge and perspectives of children and families.

As can be seen, several knowledge activities have already taken place during the course of the

evaluation. Some of the planned knowledge exchange activities originally outlined for the

evaluation, however, such as with agency partners, volunteers, F&ST staff or participants, are

still in development given the response of staff toward the findings. As aforementioned, this

evaluation has resulted in productive conversation about the future and direction of the program

as well as led to new learning about the program and about evaluation for the agency. It has

also halted knowledge exchange activities of program success or needed improvements with

other stakeholders or partners due to the reluctance of agency staff and leaders to share or

implement the recommended amendments. One planned knowledge exchange activity for the

current evaluation findings, however includes sharing the current report with agency leaders,

agency staff and the Board of Directors. After this knowledge exchange activity, it will be

decided amongst these individuals how to proceed with further knowledge sharing and with

what amendments, if any, are to be made to improve the program.

38

Works Cited Berg, B.L. (2007). Qualitative research methods for the social sciences (6th Ed.). Boston: Allyn &

Bacon. Family Service Canada. (n.d.). F&ST Program Workbook. Family Service Canada. (2004). Families and Schools Together Canada Canadian National

Evaluation of Outcomes 1996-2004 Executive Summary. Ottawa: no author. Family Service Canada. (2007). Family Service Canada Annual Report 2006-2007. Ottawa: no

author. Families and Schools Together. (2007). Families and Schools Together Canada Annual Report

2006-2007. No author. Kratochwill, T.R., McDonald, L., Levin, J.R., Scalia, P.A. & Coover, G. (2009). Families and

schools together: An experimental study of multi-family support groups for children at risk. Journal of School Psychology, 47, 245-265.

Melde, C., Esbensen, F. & Tusinski, K. (2006). Addressing program fidelity using onsite

observations and program provider descriptions of program delivery. Evaluation Review, 30 (6), 714-740.

McDonald, L., Billingham, S., Conrad, T., Morgan, A., O, N., & Payton, E. (1997). Families and

schools together (F&ST): Integrating community development with clinical strategies. Families in Society: Journal of Contemporary Human Services, 140-155.

McDonald, L. & Sayger, T.V. (1998). Impact of a family and school based prevention program

on protective factors for high risk youth. Drugs & Society, 12 (1), 61-85. Patton, M.Q. (1980). Qualitative evaluation methods. Beverly Hills, California: Sage Publications

Inc. The Family Centre. (2008). Building inclusive and respectful communities 2007-2008 Annual

Report. Winnipeg: no author. The Vanier Institute of the Family. (2006). Building emotional intelligence: Darwin reconsidered.

Retrieved April 3, 2010 from http://www.vifamily.ca/library/cft/eq.html. Wang, C. C. & Burris, M. A. (1997). Photovoice: Concept, methodology, and use for

participatory needs assessment. Health Education and Behavior, 24(3), 369-387.

39

APPENDICES

40

APPENDIX A – F&ST Logic Model and Evaluation Framework

41

42

43

APPENDIX B – Informed Consent for F&ST Families and Parent Partners

44

APPENDIX C – Interview Consent Form for F&ST Staff and Volunteers, School Personnel, Public Health Personnel and Mosaic F&ST Staff

45

APPENDIX D – Interview Schedules

INTERVIEW SCHEDULES Interviews – Current Families

1) If you were to describe F&ST to another person who didn’t know about it, how would you describe it?

2) Can you tell me a story about a positive experience you had at F&ST? 3) Can you tell me a story about a negative experience you had at F&ST? 4) What do you like best about the F&ST program in general? 5) If you could change one thing about the F&ST program in general, what would it be?

6) What have you learned from the program? Probe: What information have you taken away from the program? 7) How has the program helped you?

8) How has the program helped your family? Probe: What has the program done for you and your family? 9) Do you have any stories to share that are particularly meaningful? 10) How do you feel about this interview format (versus the survey)?

Staff/Volunteers

1) If you were to describe F&ST to another person who didn’t know about it, how would you describe it?

Probe: What would you want other people to know about the program? 2) How long have you been involved in F&ST? 3) Can you tell me a story about a positive experience you have had with F&ST? 4) Can you tell me a story about a negative experience you have had with F&ST? 5) What did you enjoy most about helping to run F&ST? 6) What did you enjoy least about helping to run F&ST? 7) What are the strengths of the program in general? 8) If you could change one thing about the program in general, what would it be? 9) How well do you feel that the things taught in training are happening as they are supposed to? 10) How has the program contributed to your agency? Probe: How does the program align with your agency mission or goals? 11) How has the program contributed to the lives of the families involved? Probe: What has the program done for the children and their families involved? 12) What have you personally learned from the program? Probe: What information have you taken away from the program?

46

13) Do you have any success stories to share? 14) How do you feel with this interview format?

Parent Partners

1) If you were to describe F&ST to another person who didn’t know about it, how would you describe it?

Probe: What would you want other people to know about the program? 2) How long have you been involved in F&ST? 3) Can you tell me a story about a positive experience you have had with F&ST? 4) Can you tell me a story about a negative experience you have had with F&ST? 5) What did you enjoy most about helping to run F&ST? 6) What did you enjoy least about helping to run F&ST? 7) What are the strengths of the program in general? 8) If you could change one thing about the program in general, what would it be? 9) How well do you feel that the things taught in training are happening as they are supposed to? 10) How has the program contributed to the lives of the families involved? Probe: What has the program done for the children and their families involved? 11) What have you personally learned from the program? Probe: What information have you taken away from the program? 12) Do you have any success stories to share? 13) How do you feel with this interview format (versus the survey format)?

Teachers/School Personnel

1) If you were to describe F&ST to another person who didn’t know about it, how would you describe it?

Probe: What would you want other people to know about the program? 2) How long have you been involved in F&ST? 3) Can you tell me a story about a positive experience you have had with F&ST? 4) Can you tell me a story about a negative experience you have had with F&ST? 5) What did you enjoy most about helping to run F&ST? 6) What did you enjoy least about helping to run F&ST? 7) What are the strengths of the program in general? 8) If you could change one thing about the program in general, what would it be? 9) How well do you feel that the things taught in training are happening as they are supposed to? 10) How has the program contributed to the lives of the families involved?

47

Probe: What has the program done for your student(s) and their family(ies)? 11) How has the program contributed to your school? Probe: How does the program align with your school’s mission or goals? 12) What have you personally learned from the program? Probe: What information have you taken away from the program? 13) Do you have any success stories to share? 14) How do you feel with this interview format (versus the survey format)?

Public Health

1) If you were to describe F&ST to another person who didn’t know about it, how would you describe it?

Probe: What would you want other people to know about the program? 2) How long have you been involved in F&ST? 3) Can you tell me a story about a positive experience you have had with F&ST? 4) Can you tell me a story about a negative experience you have had with F&ST? 5) What did you enjoy most about helping to run F&ST? 6) What did you enjoy least about helping to run F&ST? 7) What are the strengths of the program in general? 8) If you could change one thing about the program in general, what would it be? 9) How well do you feel that the things taught in training are happening as they are supposed to? 10) How has the program contributed to the lives of the families involved? Probe: What has the program done for your student(s) and their family(ies)? 11) How has the program contributed to your agency? Probe: How does the program align with your agency mission or goals? 12) What have you personally learned from the program? Probe: What information have you taken away from the program? 13) Do you have any success stories to share? 14) How do you feel with this interview format?

Past Participants

1) If you were to describe F&ST to another person who didn’t know about it, how would you describe it?

Probe: What would you want other people to know about the program?

48

2) Can you tell me a story about a positive experience you had at F&ST? 3) Can you tell me a story about a negative experience you had at F&ST? 4) What are the strengths of the program in general? 5) If you could change one thing about the program in general, what would it be? 6) 7) How has the program contributed to your family’s life? Probe: What has the program done for you and your family? 8) Have there been any lasting changes in your family since the program? 9) Have you kept in contact with any families in the program? 10) 11) What have you personally learned from the program? Probe: What information have you taken away from the program? 12) Do you have any stories to share that are particularly meaningful? 13) How do you feel with this interview format (versus the survey format?)

Photovoice Instructions: Please take pictures that represent important and meaningful discoveries made during the program and the ways that the program has impacted you and your family. Please take pictures that show: Anything new that you or your family have learned through the program Anything changes that have taken place in your family as a result of the program Anything you and your family feel is special to you about the program Anything you and your family would want others to know about the program

1) Where was this picture taken? 2) Why did you take this picture? 3) What was happening in this situation? Probe: what would you want people to know about this situation/picture? Probe: How does this picture connect to the F&ST program? 4) Was this situation positive or negative? Why? 5) How did you feel about participating in Photovoice? 6) How would you feel about participating in Photovoice again? 7) What did you enjoy most about participating in Photovoice? 8) What would you change about Photovoice?

Please also ask interview questions for current F&ST families

49

Appendix E – Recruitment Flyer for Potential Photovoice Participants

50

APPENDIX F – Informed Consents for Photovoice Participants

51

APPENDIX G – Information and PowerPoint Introduction to Photovoice for Families

52

53

54

55

56

57

58

APPENDIX H – Photovoice Focus Group Recruitment Flyer

59

APPENDIX I – Photovoice Focus Group Informed Consent Form

60

APPENDIX J – One Page Summary for Poster Conference at CHEO

61