Top Banner
A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions by Nadine Reid A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Institute of Health Policy, Management and Evaluation University of Toronto © Copyright by Nadine Reid 2017
169

A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

Oct 09, 2020

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions

by

Nadine Reid

A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy

Institute of Health Policy, Management and Evaluation University of Toronto

© Copyright by Nadine Reid 2017

Page 2: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

ii

A Realist Evaluation of Family Navigation in Youth Mental Health

and Addictions

Nadine Reid

Doctor of Philosophy

Institute of Health Policy, Management and Evaluation

University of Toronto

2017

Abstract

In Canada today, many families of youth with mental health and/or addiction concerns are

struggling to access the care they need. The Family Navigation Project is a service affiliated with

Sunnybrook Health Sciences Centre in Toronto, Ontario, which aims to provide needs-based,

family-centred system navigation to families of youth aged 13 to 26 with mental health and/or

addiction concerns. The current study is a Realist Evaluation of the Family Navigation Project.

The objectives of this study were a) to describe the population being served by the Family

Navigation Project; b) to develop a conceptual framework for family navigation and a program

theory for the Family Navigation Project; and c) to test the program theory, and refine it based on

the results. This multi-phase, mixed methods study applied a Realist Evaluation framework and a

cross-sectional methodological design in which both quantitative and qualitative data were

collected through an online survey package in order to evaluate the sample characteristics;

perception of navigation; the impact of perceived experience on family empowerment, family

quality of life, and service satisfaction; and the influence of context. Data was collected from a

convenience sample of 134 families seeking care on behalf of youth, who were registered with

the Family Navigation Project at the time of the study. Descriptive, inferential and qualitative

Page 3: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

iii

analyses were performed. Results indicated that the Family Navigation Project reached its target

population in this sample; that families in this sample were highly satisfied with the services they

received; that most families in this sample perceived care to be accessible, continuous, and

family-inclusive; that this perceived experience significantly influenced family empowerment,

family quality of life, and service satisfaction; and that both individual and systemic-level

contexts influence experience and outcomes to varying extents. The conceptual framework and

program theory were subsequently refined. Applications, contributions and limitations are noted.

Page 4: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

iv

Acknowledgments

I would like to first express my sincere gratitude to the families involved in this study for sharing

their stories. I would also like to thank the staff at the Family Navigation Project for their

ongoing enthusiastic participation and feedback. Specifically, I would like to thank Dr. Anthony

Levitt for introducing me to the topic of navigation and including me in the development of the

Family Navigation Project many years ago; and for his ongoing support since then.

To my supervisors, Dr. Rhonda Cockerill and Dr. Janet Durbin, thank you for your consistent

(re)direction over the last four years, and for patiently and reliably keeping my feet (and ideas)

on the ground.

Finally, thank you to my husband and family, who always help to keep things in perspective; and

to Ruby, for cuddle breaks, bottomless laughs, and daily walks in the park.

Page 5: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

v

Table of Contents

Acknowledgments.......................................................................................................................... iv

Table of Contents .............................................................................................................................v

List of Tables ...................................................................................................................................x

List of Figures ............................................................................................................................... xii

List of Appendices ....................................................................................................................... xiii

Chapter 1 Introduction .....................................................................................................................1

Description of the current state ...................................................................................................1

1.1 Navigation: A potential solution ..........................................................................................4

1.1.1 Navigation in mental health and addictions .............................................................5

1.2 The Family Navigation Project ............................................................................................9

1.2.1 Background and project development .....................................................................9

1.2.2 Project description .................................................................................................10

Chapter 2 Theoretical and Conceptual Frameworks ......................................................................14

Theoretical evaluation framework: Realist evaluation .............................................................14

2.1 Theory-building process ....................................................................................................16

2.1.1 Program theory.......................................................................................................18

2.2 Conceptual framework .......................................................................................................19

2.2.1 Context ...................................................................................................................19

2.2.2 Mechanisms ...........................................................................................................20

2.2.2.1 Accessibility ............................................................................................20

2.2.2.2 Continuity of care ....................................................................................21

2.2.2.3 Family involvement .................................................................................21

2.2.3 Outcomes of interest ..............................................................................................22

Page 6: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

vi

2.2.3.1 Family empowerment ..............................................................................23

2.2.3.2 Family quality of life ...............................................................................24

2.2.3.3 Service satisfaction ..................................................................................25

Chapter 3 Research Questions and Rationale ................................................................................27

Research questions ....................................................................................................................27

3.1 Hypotheses .........................................................................................................................27

3.2 Objectives ..........................................................................................................................28

3.3 Rationale for the current study ...........................................................................................29

Chapter 4 Measurement .................................................................................................................30

Specification of variables ..........................................................................................................30

4.1 Context variables ...............................................................................................................30

4.2 Mechanism variables .........................................................................................................30

4.3 Outcome variables .............................................................................................................31

4.3.1 Family empowerment ............................................................................................31

4.3.2 Family quality of life .............................................................................................32

4.3.3 Service satisfaction ................................................................................................33

4.4 Qualitative measures ..........................................................................................................34

4.5 Measurement validity.........................................................................................................35

Chapter 5 Methods .........................................................................................................................37

Design overview .......................................................................................................................37

5.1 Study setting.......................................................................................................................37

5.2 Ethics..................................................................................................................................37

5.3 Sample................................................................................................................................37

5.4 Data collection ...................................................................................................................38

5.4.1 Survey methods ......................................................................................................38

Page 7: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

vii

5.4.1.1 Pilot phase overview................................................................................39

5.4.1.2 Response rates .........................................................................................39

5.4.2 Chart review ...........................................................................................................41

5.5 Sample size and power.......................................................................................................41

5.6 Data storage .......................................................................................................................42

5.7 Overview of analytical approach .......................................................................................42

Chapter 6 Quantitative Analysis ....................................................................................................44

Overview of quantitative approach ...........................................................................................44

6.1 Importing and coding of data .............................................................................................44

6.2 Data quality ........................................................................................................................45

6.3 Descriptive analyses...........................................................................................................45

6.3.1 Context variables ...................................................................................................45

6.3.1.1 Demographics ..........................................................................................46

6.3.1.2 Mental health and addiction characteristics ............................................48

6.3.1.3 Previous service use ................................................................................51

6.3.1.4 Reasons for contact..................................................................................52

6.3.2 Mechanism variables .............................................................................................53

6.3.2.1 Accessibility ............................................................................................53

6.3.2.2 Continuity of care ....................................................................................54

6.3.2.3 Family involvement .................................................................................56

6.3.3 Outcome variables .................................................................................................56

6.3.3.1 Family empowerment ..............................................................................57

6.3.3.2 Family quality of life ...............................................................................60

6.3.3.3 Service satisfaction ..................................................................................62

6.4 Correlational analysis.........................................................................................................64

Page 8: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

viii

6.4.1 Overview of approach ............................................................................................64

6.4.2 C-M and C-O dyads: Selection of covariates for inferential modelling ................65

6.4.3 M-M, M-O and O-O dyads: Satisfaction of modeling assumptions ......................71

6.4.3.1 M-M dyads ..............................................................................................71

6.4.3.2 M-O dyads ...............................................................................................72

6.4.4 Implications for modeling ......................................................................................72

6.5 Factor analysis ...................................................................................................................73

6.6 Inferential modelling ..........................................................................................................75

6.6.1 Overview of statistical approach ............................................................................76

6.6.2 Testable hypotheses ...............................................................................................78

6.6.3 Models of family empowerment ............................................................................80

6.6.3.1 FES Family subscale ...............................................................................81

6.6.3.2 FES Service-seeking subscale .................................................................83

6.6.4 Model of family quality of life...............................................................................85

6.6.5 Models of service satisfaction ................................................................................87

6.6.5.1 Inverse NAVSAT total score...................................................................87

6.6.5.2 Inverse satisfaction with referred services (SRS) score ..........................89

Chapter 7 Qualitative Analysis ......................................................................................................93

Overview of qualitative approach .............................................................................................93

7.1 Descriptive and thematic analyses .....................................................................................95

7.1.1 Context ...................................................................................................................95

7.1.1.1 Theme 1: You can’t force someone to get well .......................................95

7.1.1.2 Theme 2: You can’t navigate to services that don’t exist........................97

7.1.1.3 Theme 3: Existing services lack accessibility and continuity of care .....99

7.1.1.4 Theme 4: Privacy, consent and capacity legislation ..............................101

Page 9: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

ix

7.1.2 Mechanisms .........................................................................................................103

7.1.2.1 M(resources) ..........................................................................................103

7.1.2.2 M(reasoning) .........................................................................................107

7.1.2.3 The link between resources and reasoning ............................................110

7.1.3 Outcomes .............................................................................................................111

7.1.3.1 Positive outcomes ..................................................................................112

7.1.3.2 Negative outcomes ................................................................................114

Chapter 8 Discussion ...................................................................................................................117

Results as per the research questions, conceptual framework and program theory ................117

8.1 Research question 1 .........................................................................................................117

8.2 Research question 2 .........................................................................................................120

8.3 Research question 3 .........................................................................................................122

8.4 Results in relation to the conceptual framework and program theory .............................125

Chapter 9 Conclusions .................................................................................................................129

Limitations, mitigations and contributions .............................................................................129

9.1 Study limitations and mitigations ....................................................................................129

9.2 Contributions....................................................................................................................132

References ....................................................................................................................................134

Appendix A Themes from the Family Navigation Project’s Focus Groups ................................142

Appendix B Logic Model and Conceptual Framework ..............................................................147

Appendix C Measurement Summary ...........................................................................................150

Page 10: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

x

List of Tables

Table 1. Components of family navigation

Table 2. Hypotheses by research question and dependent variable

Table 3. Descriptive statistics for continuous current program use variables

Table 4. Frequency statistics for demographic variables

Table 5. Descriptive statistics for continuous mental health and addiction characteristic variables

Table 6. Frequency statistics for mental health characteristic variables

Table 7. Frequency statistics for addiction characteristic variables

Table 8. Frequency statistics for service use characteristic variables

Table 9. Frequency statistics for reasons for contact variable

Table 10. Descriptive statistics by item for the “Accessibility scale”

Table 11. Descriptive statistics by item for the “Continuity of care scale”

Table 12. Descriptive statistics for the single item “Family involvement”

Table 13. Descriptive statistics by item for the “FES Family subscale”

Table 14. Descriptive statistics by item for the “FES Service-seeking subscale”

Table 15. Descriptive statistics for the “m-BCFQoLS scale”

Table 16. Descriptive statistics for the “NAVSAT total score scale”

Table 17. Results of a correlational analysis between select C-M and C-O dyads

Page 11: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

Table 18. Summary of covariates and predictors for modeling by outcome

Table 19. Spearman’s rho correlation coefficients for mechanism and outcome variables

Table 20. Results of a Principal Components analysis for mechanism variables

Table 21. Descriptive statistics for the Bartlett Factor Score for “navigation mechanism”

Table 22. Correlation coefficients for the Bartlett Factor Score with outcomes

Table 23. Dependent variables and corresponding hypotheses

Table 24. Covariates selected for inclusion in modeling by outcome

Table 25. Parameter estimates for the dependent variable “FES Family score”

Table 26. Parameter estimates for the dependent variable “FES Service-seeking score”

Table 27. Parameter estimates for the dependent variable “m-BCFQoLS total score”

Table 28. Parameter estimates for the dependent variable “Inverse NAVSAT total score”

Table 29. Parameter estimates for the dependent variable “Inverse satisfaction with referred

service (SRS) score”

Table 30. Omnibus Test of Model Fit results by dependent variable

Table 31. Generalized linear model equations by dependent variable

Table 32. M-M dyad combination by theme

Page 12: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

xii

List of Figures

Figure 1. Continuum of service delivery models with examples of existing programs

Figure 2. Word cloud depicting word frequency in all phrases coded as an outcome

Page 13: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

xiii

List of Appendices

Appendix A: Themes from the Family Navigation Project’s Focus Groups

Appendix B: Logic Model and Conceptual Framework

Appendix C: Measurement Summary

Page 14: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

1

Chapter 1 Introduction

Description of the current state

In Canada today, three out of four youth struggling with mental health and/or addiction (MHA)

concerns do not receive the care they need (Mental Health Commission of Canada [MHCC],

2015b). Mood and anxiety disorders rates among youth are rising, and suicide remains the

second leading cause of death at a rate that has not decreased significantly in the last decade

(MHCC, 2015a). Of all age groups, youth aged 15-24 are the most likely to experience these

types of health concerns and yet least likely to access services (Patton, 2007). Recently released

data from the Canadian Institute for Health Information [CIHI] indicates that families of youth

aged five to 24 with mental health disorders are increasingly seeking care in hospital settings due

to the unavailability or inaccessibility of community-based services; youth emergency

department (ED) visits and inpatient admissions for mental health reasons have increased by

56% and 47%, respectively, in a 10-year period (CIHI, 2017).

There are a variety of explanations behind these statistics, many of which are associated with

systemic challenges in the amount of awareness, knowledge, capacity, integration, coordination,

accessibility and accountability for youth MHA treatment (Institute for Clinical Evaluative

Sciences [ICES], 2017; MHCC, 2012, 2015b, 2017; Ministry of Health and Long Term Care

[MOHLTC], 2011). There is a particular lack of capacity for the specialized treatment of

complex conditions involving one or more MHA concerns (Canadian Mental Health Association

[CMHA], 2013; CIHI, 2013). The impact of this gap is significant as MHA concerns frequently

co-occur (Kessler, et al., 2005, 2012; Pearson, Janz, & Ali, 2013). The fragmented system that

results from insufficient capacity means that it is often extremely difficult to identify and access

the right doors and subsequent pathways to appropriate care (MHCC, 2012, 2015a; MOHLTC,

2011). In the field of youth MHA concerns, it is often the family (i.e. parent or guardian) who

facilitates the interaction with the health care system and negotiates access to resources on the

youth’s behalf, yet family-centred services are uncommon (Curtis & Singh, 1996). Families

seeking care for youth in crisis are often forced to try a wide range of resources before finding an

appropriate match, and frequently encounter multiple long wait lists along the way that can lead

to a closed window of opportunity before families can find the help they need (MHCC, 2012,

Page 15: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

2

2015a; MOHLTC, 2009, 2011). A concerning byproduct of this process is reduced capacity in

the service system over the long-term (Children’s Mental Health Ontario [CMHO], 2016a;

MHCC, 2017).

Delays in access to care can have serious, long-term health, social, and economical

consequences. Nearly 70% of MHA concerns have their onset in adolescence, and without

timely and appropriate care, these concerns become lifelong struggles that impact individual and

family health, quality of life, and productivity, beyond placing a significant burden on the health

care system (MHCC, 2017; Statistics Canada, 2006). Timing is critical when seeking care and

intervening for this vulnerable population as the developmental period from adolescence to

young adulthood is a cornerstone in which individuals establish long-term patterns of behaviour

and make life decisions that can have significant impacts throughout the lifespan (Kessler, et al.,

2005, 2012). A recent report by the Mental Health Commission of Canada indicates that the

economic burden of mental health problems exceeds $50 billion per year; and lifetime economic

costs of childhood mental health problems are estimated at approximately $200 billion in Canada

(MHCC, 2017; Smith & Smith, 2010).

The provincial health care system is difficult to navigate for patients, families, caregivers,

providers and policymakers alike (MOHLTC, 2011). MHA services include both public and

private services; community agencies and hospital services; services with different eligibility

criteria, age restrictions and residency requirements; and services provided by many different

designated professions that may or may not be regulated and/or funded by up to four different

provincial ministries. As a consequence, service providers and professionals working in one

service sector often lack knowledge of other sectors offering services and resources that are still

relevant to a family’s multidimensional needs, making it difficult for a family to obtain the

comprehensive basket of services they require (MHCC, 2012).

For families of youth with suspected MHA concerns, equally challenging is the question of

knowing what to do and where to go for help in the face of a suspected problem that may be

apparent and yet difficult to articulate. Research suggests that families are continuing to look to

their family doctors or pediatricians for a first source of help (CIHI, 2017; MOHLTC, 2011).

However, this is typically only the first stop of many; in most cases, families make contact with

multiple agencies or providers before they find what they need, and they learn about treatments

Page 16: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

3

and professionals informally as they go along. Moreover, while some families feel fully engaged

in their youth’s treatment, others may feel excluded. This may reflect the particular treatment

philosophy or bias of the provider, the youth’s choice, involvement of other service sectors, or

limiting interpretations of current privacy and consent legislation. Because there is no age of

consent in Canadian health care law, youth may be deemed capable of refusing treatments in

spite of certain need, and parents may not be informed or recruited to be involved in the youth’s

treatment pathway (Ontario Hospital Association [OHA], 2016). Generally, health care providers

will work with the youth and their family to reach a consensus on an agreeable course of action

for all parties involved. However, in some cases, consensus is not reached, and this is a situation

that occurs disproportionately in the management of MHA concerns.

Even for physicians, navigating such a complex health care system is challenging. American

families have reported that, although they were initially able to connect with a mental health

provider, often there was no clear pathway from that point onwards (Lazear, Worthington, &

Detres, 2004). Although the Canadian system is largely incomparable, here too family physicians

are most often the first and preferred point of contact with the health care system, and are also

expected to provide access to primary comprehensive mental health care and serve as

gatekeepers to specialty services (MOHLTC, 2011; Pautler, 2005). In fact, Ontario Family

Health Teams (FHT) were designed to serve as “patient navigators,” and so “care navigators”

were an explicitly recommended staffing option in the FHT collaborative practice guidelines

(although no studies evaluating how many FHTs have chosen to employ this position to date and

whether or not it is effective are known to exist) (Conference Board of Canada, 2014; MOHLTC,

2005).

The burden of suffering for families can be immense; parents often speak of feeling adrift and by

themselves. A 2006 report from the Canadian Mental Health Association voiced families’ desire

for better information and education, peer support to share experiences and coping skills, and

opportunities for caregiver relief, such as respite (CMHA, 2006). Today, for most families,

resources still remain extraordinarily difficult to navigate, despite repeated calls for navigation

services since then on both the provincial and national stage (MOHLTC, 2009, 2011).

Page 17: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

4

1.1 Navigation: A potential solution

Navigation and related terms such as patient navigation, care navigation, and in the current

scenario, family navigation, is a concept that has gained traction over the last two decades and is

now increasingly emerging in a range of health care settings (Freeman & Rodriguez, 2011).

Navigation is often categorized as a care coordination intervention in that navigators are thought

to facilitate standardized care coordination activities that support physicians’ essential care tasks

typical of the general patient population (McDonald, et al., 2007).

Although navigation tends to get grouped with care coordination (and particularly case

management, which is a type of care coordination intervention), navigation-specific literature,

albeit limited, does suggest that navigation can and should be distinguished from care

coordination and case management. Whereas care coordination is defined by a standardized set

of activities, proponents of navigation emphasize that navigation is distinguished from care

coordination by its unique patient-focused approach. The central focus is helping patients to

overcome perceived barriers, similar to a case management model of care (Longest & Young,

2000; McDonald, et al., 2007). By its champions, professional navigator roles are envisioned as

extending beyond even the role of case managers, who are considerably more task-oriented and

typically manage appointment coordination and adherence. Instead, navigators favour a

comprehensive social model of patient management that values humanization of the care

trajectory and empowerment of the patient and family to overcome their perceived barriers

(Fillion, et al., 2006). However, because of this, navigation role descriptions are often highly

context-specific, which contributes to the lack of consensus and consistency in the literature

(Pedersen & Hack, 2010).

Navigation services have their roots in cancer care (breast and cervical cancer, in particular),

where programs were predominantly focused on overcoming barriers among disadvantaged

patients in order to ensure equitable access to adequate care for all, regardless of gender, ethnic

or socioeconomic status (Robinson-White, Conroy, Slavish, & Rosenzweig, 2010). Dr. Harold

Freeman is widely known to have first coined the term “patient navigation” when thinking about

“a metaphor for what patients have to do to negotiate the medical system; that of being on a

small boat in the south seas when you can see an island in the distance you want to get to but

there are rocks in between, and if you hit them, you’ll sink and possibly die. But let’s say there’s

Page 18: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

5

a navigator on board who can chart the course and get you there safely.” He goes on to

emphasize the value navigators hold for patients and systems alike because “it’s very easy to get

lost in the impersonal health care system, but navigators are like problem solvers. They

coordinate all the disjointed elements and move patients through fast and more efficiently”

(Freeman, 2011).

Within Canada, in 2004, a formal evaluation of pilot projects in breast cancer navigation by

Cancer Care Nova Scotia found consistent benefits of navigation for patients and families that

included providing emotional support, preparing patients for their cancer journey, referrals to

appropriate health professionals, increasing patient knowledge about cancer, helping to

coordinate appointments, referral to community supports, assisting with the logistics of getting to

cancer centres and finding sources of funding for medications and supplies (Corporate Research

Associates Inc., 2004).

1.1.1 Navigation in mental health and addictions

Care coordination and related interventions have had a place in the field of MHA concerns for

decades, but since they operate within a fragmented system where silos and barriers span the

wide range of sectors and providers involved, any one provider faces inherent impediments to

meeting the extensive and diverse service needs of the population. Incompetent navigation of

these barriers is associated with poor patient outcomes and inappropriate health care utilization,

but a catch-22 exists when the results of disparate coordination efforts further perpetuate

fragmentation (McDonald, et al., 2007). Here, it is also important to reiterate that coordination is

not navigation; coordination interventions are driven by a pre-defined set of services, whereas

navigation helps patients overcome their perceived barriers. This approach is likely to be both

more efficient and effective when it comes to finding resources for MHA problems, which are

known to manifest via highly unique and individualized pathways and require similarly

individualized responses.

Undoubtedly, there is an intuitive place for navigation services in addressing MHA concerns,

and even more so for youth and families. The gap has not gone unrecognized. Ontario’s 2011

mental health and addiction strategy, Open Minds, Healthy Minds, states that “mental health and

addictions services are fragmented, spread across several ministries and offered in a variety of

care settings;” and that “families struggle to navigate services and get the right support for their

Page 19: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

6

children and youth.” With over $250 million in new funding over three years, one of the key

youth priorities moving forward was the provision of “fast access to high quality service,”

including specific initiatives meant to “improve public access and help children, youth and

families find the right kind of services” and “provide supports in select communities for families

to navigate the system” (MOHLTC, 2011).

Since then, a handful of examples of MHA navigation initiatives emerged across Canada, such as

British Columbia’s Sooke Navigator Project and F.O.R.C.E. Society for Kids’ Mental Health,

Halton (Ontario) Healthcare Services’ Navigator Program, and COAST Navigation in Niagara,

Ontario (McPhee, Syed, Nunes, & Mobilizing Minds Research Group, 2012; Roberts & Schmidt,

2012). More recently, St. Joseph’s Centre in Toronto, Ontario launched a Family Navigation

Program. While this list is not exhaustive, the following examples are self-defined navigation

programs that share an emphasis on overcoming patient-perceived barriers to accessing

appropriate MHA care. However, it is important to note the extensive variation and lack of

standard practices in the way navigation programs are implemented, organized and delivered.

Because of this, while a fixed list of components cannot be neatly attributed to each category,

one can more generally describe these models to aid in understanding the service continuum.

Numerous jurisdictional reviews and conversations with experts in the field have led to the

conclusion that currently in Canada, navigation programs exist along a continuum that varies in

terms of which services are offered, to what extent, and to whom. In general, as you move along

the continuum toward programs with more to offer to more people, one expects increased

accessibility, increased client engagement, and increased continuity of care (Figure 1).

Online or tele-help

• ConnexOntario

• Kids Help Phone

• Mental Health Helpline

Interorganizational referral networks

• C.O.A.S.T. Niagara

• Halton Mental Wellnes Navigator Program

• Contact Hamilton

Navigation representative

• FHT Navigators

• St. Joseph's Family Navigation Program

• Hamilton's Child and Youth Aboriginal Navigator

• CMHA BC Community Navigator

• SickKids Mental Health Patient Navigator

Navigation team

• Sooke Navigator Project

ContinuityEngagementAccessibility

Page 20: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

7

Figure 1. Continuum of service delivery models with examples of existing programs

The most basic delivery of navigation services exists by one-dimensional online or tele-help

resources. ConnexOntario is an excellent example of this model. It can be accessed online or

over the phone, which is convenient, but may or may not allow you access to a live agent, which

is not ideal due to the highly complicated nature of the MHA service systems. Accessibility of

information is thus limited as it only provides help-seekers with access to some information

about some of the services available in their area, along with a contact number. There is no

follow up or long-term engagement, and particularly for youth with MHA concerns, the

limitations of this model quickly becomes clear. For example, the platform can only search for

resources for one presenting problem; if a youth has an eating disorder and depression, or an

anxiety disorder and substance use problems, there are no results because the MHA service

systems are separate and the platform does not account for this. Yet, the co-occurrence of MHA

problems is widely known. Moreover, since ConnexOntario is a publicly-funded initiative, it

only returns search results for publicly-funded resources. This could be problematic due to the

number of private service providers, particularly for substance use treatment programs.

The second type of navigation exists within a predefined interorganizational network; structured

referral processes allow programs to facilitate inter-organizational client transfers when certain

needs cannot be accommodated by a partner program. Halton Healthcare’s Mental Wellness

Navigator Program is an example of this in that navigation services are provided only to youth

up to the age of 18 who live in the Halton catchment area and were referred because they are

already clients of Halton Healthcare. Here, they are engaged in care planning and some degree of

continuity is experienced, but once the youth turns 19, they become the responsibility of another

Ministry and are no longer eligible for the program. Contact Hamilton is another example of a

program which provides youth and families with central intake and access to a number of partner

programs in the Hamilton area. The immediate improvement over online or tele-help programs is

access to a live agent and engagement in personalized assessment. However, in both programs,

there may be some initial follow-up while connecting families with a service, but once

connected, the relationship is terminated.

The third modality is a designated navigation official or representative. This is an individual who

functions as a designated navigator within a team setting and generally aims to work with

patients throughout their help-seeking journey. For example, as mentioned earlier, Ontario FHTs

Page 21: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

8

were encouraged to include a patient navigator in their collaborative team practices. Similarly,

many cancer care programs, including the Odette Cancer Centre at Sunnybrook Health Sciences

Centre, have a dedicated navigator on staff. These individuals may be trained professionals or

peers with lived experience; either way, they tend to have a nuanced understanding of the service

system and as such, are often able to offer clients personalized care planning. British Columbia’s

branch of the Canadian Mental Health Association employs both a Community Navigator,

dedicated to finding and connecting clients with housing and social supports, and a Peer

Navigator, who focuses on providing and connecting clients to peer supports. In 2016, St.

Joseph’s Health Centre launched their Family Navigation Program, which pairs family members

with social workers who guide them through their time at the hospital and provide information,

education, counselling and support groups, and help to facilitate connections to community-

based resources in order to ensure continued support post-hospital.

At the far end of the spectrum are navigation teams, where clients should theoretically

experience the greatest degree of accessibility, engagement and continuity of care. These are

collaborative teams of designated navigation personnel dedicated to helping clients understand,

identify and access a wide range of appropriate resources to meet the range of their perceived

needs. They have the mandate and capacity to stay involved with their clients long-term,

providing follow-up and responsiveness to change as needed. The Sooke Navigator Project

(2007) is perhaps Canada’s most well-known example of the potential of navigation teams.

Developed in British Columbia and embedded within a local family service organization, the

primary intent was to improve inter-provider efficiency and capacity within the primary care

service system by providing low-barrier access, assessments, collaborative service planning,

respectful communication, referrals, tracking and follow-up for anyone who asked, including but

not targeted to youth and families. The Sooke Navigator Project is one of the few identifiable

examples of independent navigation team initiatives, and evaluative research suggests the model

effectively improved understanding of, access to and connectivity of community services

(Anderson & Larke, 2009). Indicative of its face validity, the model is being implemented at

three additional sites in British Columbia.

Overall, the range of service delivery models appears to differ significantly in several important

ways, including but not limited to the engagement method, provider type, location and

organization of services, and intensity and length of contact. These are defining organizational

Page 22: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

9

characteristics that strongly depend upon the needs of a community and the program inputs,

which will trigger different mechanisms and outcomes for families seeking care for their youth

with MHA concerns.

1.2 The Family Navigation Project

The Family Navigation Project (FNP) is a highly unique and innovative family navigation team

initiative that launched in Toronto, Ontario in 2014 (Roberts & Schmidt, 2012). The Primary

Investigator (PI) for this study was employed as the FNP’s Project Manager from 2011 to 2013,

and as such, participated in much of the FNP’s design, development and implementation. No

existing programs are known to be adequately comparable to the FNP. As discussed in the

previous section, extensive jurisdictional and literature reviews confirm that what potential

comparators do exist vary very significantly in context, target population, goals, and actual

services offered. Some of the key features that distinguish the FNP from its comparators are its

target population (families of youth and transitional-aged youth with mental health and/or

addiction needs); its defining family-centred approach; its dedicated capacity (with seven full-

time Navigators), having navigated for over 1850 families to date; its service organization (e.g.

phone and email based, flexible hours, not for profit status).

1.2.1 Background and project development

In 2009, three families of youth treated at Sunnybrook Health Sciences Centre’s (SHSC)

inpatient youth psychiatry unit independently approached the department to express frustration

with how their children’s care plans were managed; each family had been discharged from

inpatient care with unclear follow-up or connection to community resources or supports. As a

site-based quality improvement project, three focus groups were conducted by an independent

consultant with parents of youth cared for on the unit to better understand the families’

perspectives on what was needed to improve their help-seeking and care experiences.

Strong themes emerged from the focus group data, which were then used to inform the FNP’s

business case (see Appendix A for a summary of focus group themes, which were included in the

business case; and for the full business case, refer to Roberts & Schmidt, 2012). Families

reported that having comprehensive, objective information about treatment options helps,

especially understanding the different treatment approaches, availability and a sense of the way

Page 23: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

10

in which a particular provider works and the likelihood of their being a good match for the child

and family. Parents also reported that it is helpful to hear other families’ advice and experience.

Most commonly however, parents spoke of the immeasurable value of having someone who is

knowledgeable, compassionate and committed as a guide who is prepared to “get in the boat”

with the family and stay with them throughout their journey. Indeed, this is reminiscent of Dr.

Freeman’s original metaphor.

As a whole, the data suggested that in the field of youth MHA services, pockets of knowledge

and awareness of available services are isolated from each other, a result of which is that families

experience many gaps in care. Interviews were then conducted with a wide range of families,

public and private service providers for both mental health and addictions (i.e. general

practitioners who are often first points of contact; psychiatrists; hospital and- community-based

programs and agencies; residential programs in Canada and the United States) in order to collect

as many perspectives as possible as to what the problem is, and what would be the most efficient

and effective solutions.

1.2.2 Project description

With an indication from families of what would be most helpful to them, a team of clinicians,

consultants, parents and research personnel in Toronto, Ontario developed an original model for

a lived-experience-driven, privately funded, non-profit, relationship-based navigation service for

families of youth aged 13 to 26 with MHA concerns. The program is affiliated with Sunnybrook

Health Sciences Centre, and officially launched in June 2014.

The FNP purports to address identified service gaps by partnering families with navigators who

are skilled, knowledgeable and experienced mental health professionals able to provide families

with needs-specific and family-centred information in order to facilitate identification of and

access to the appropriate services and resources for the identified youth client, as well as the

entire family, as needed. On an individual (i.e. family)-level, navigators offer low-barrier access

to highly individualized, family-centred, comprehensive perceived needs assessments;

informational support; collaborative resource and service planning; referrals; care coordination;

and follow up and support throughout to ensure that the process is working. If not, navigators re-

evaluate perceived needs, barriers and the service plan in order to provide the family with a more

currently appropriate solution. The program is driven by the underlying assumption that MHA in

Page 24: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

11

youth are experienced by the entire family, and consequently, that a family-centred coordinated

and continuous care plan in which families are engaged, empowered, and supported, will lead to

improved satisfaction with the service and with help-seeking in general, and improved family

quality of life.

At the systems-level, the FNP liaises with other resource providers and system stakeholders to

help match resources to need and to build relationships over time, which may help to facilitate

system coordination, and eventually capacity. A full-time Director of Strategy and Partnerships

cultivates relationships throughout the health care community to improve integration and

awareness. Educational events and presentation to a wide range of stakeholders help raise

awareness about the resources available, and empower families with the tools to recognize

problems and potential solutions. The FNP is also continuously developing an in-house live

database of resources, programs and services for youth MHA treatment, which includes nuanced

service details and data on treatment outcomes so families can be directed toward the most

innovative, successful approaches and programs for a particular situation; and so evidence of

unmet needs can be identified and used to promote informed, efficient distribution of system

resources.

Logistically, the “program team” consists of an intake worker, seven full-time navigators, several

administrators and a growing research staff, all of whom are led by a Medical Director. Access to

the program is primarily through an intake email and phone line. Similarly, the program is

primarily phone- and email-based, although onsite meeting facilities are available should this be

a family’s preference. For families who phone or email central intake, an assessment process is

quickly initiated. Initial demographic information, reported MHA concerns, and reasons for

contacting the program (among other variables) are collected, and clients are assessed by an

intake worker so they can be paired with a Navigator particularly suited to that family’s needs.

The assigned Navigator then completes a more comprehensive, family-based needs-assessment

and commits to navigating for that family from that point onward, as long as needed. Although

client status will automatically update to discharged if there has not been contact within a certain

period of time, families are encouraged to return to the program, as needed, at any point in time.

Page 25: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

12

Following extensive discussions with the program team, family navigation was determined to be

defined by the key components outlined and defined in Table 1. The specific process by which

these components were determined is discussed in Chapter 2.

Table 1. Components of family navigation

Component Definition

Accessible expertise

Services are organized to respond to families' needs; phone and

email-based, extended and flexible hours, meeting space; medical

consultation and supervision available; Navigator experience and

expertise

Family education

Navigators provide families with information and resources aimed

at improving their understanding of the problem itself, and the

roles of the family, Navigators, and health care system in the

recovery process

Resource assessment and

information sharing

Navigators provide appropriate and sufficient information on

which resources are available

Resource matching Navigators identify the resources that best meet the family's

identified needs

Referrals Navigators directly connect families and facilitate relationships

with identified resources or services

Connections to peer support Navigators provide families with connections to support networks

of peers with lived experience

Family-based perceived needs

assessment

Navigators discuss and determine the youth's and family's

perceived needs and perceived barriers to care

Family-based collaborative care

planning

Navigators engage and work closely with families to develop care

plans with clear steps and supports that meet their perceived needs

and overcome perceived carriers

Consistent family engagement Navigators consistently engage with families in all steps

throughout the process

Information dissemination and

exchange

Navigators ensure continuous and current information exchange

across care settings so families perceive transitions as seamless

Ongoing follow up and response to

changing needs

Navigators communicate with families regularly to continuously

monitor progress and adjust course as necessary throughout the

process

Relationship and trust-building Navigators strive to develop a relationship and build rapport with

families by providing care and communicating in a professional,

Page 26: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

13

respectful and responsive manner

Facilitate service-system

relationships*

Navigators visit and build relationships with service providers in

the system

Promote evaluation and

accountability*

Navigators evaluate their own service and other resources in order

to promote efficiency and effectiveness in the system through

evidence-based care

Promote education and awareness

for youth MHA*

The Family Navigation Project conducts research and shares

findings at educational events targeted at a wide range of

stakeholders and settings

Advocate for youth MHA and

navigation services*

The Family Navigation Project advocates for its cause on a variety

of platforms using primary data

Competency training* Navigators must be highly skilled and experienced mental health

and/or addiction professionals.

*Indicates service-system interaction; this level is beyond the scope of the current study, which is limited to family-

service interaction.

Page 27: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

14

Chapter 2 Theoretical and Conceptual Frameworks

Theoretical evaluation framework: Realist evaluation

The FNP’s design and development aligned with a traditional realist synthesis process in that it

began with highly contextualized data from a wide range of sources to identify and understand

the gaps in care before soliciting feedback regarding the most efficient and effective way to

mitigate these gaps (Pawson & Tilley, 1997). Research was collected over two years and

incorporated into a constantly evolving model of care. Similar to the FNP’s development

process, an evaluation framework for the FNP that would be highly adaptive, realistic, context-

specific, and stakeholder-driven was determined by the PI (in consultation with the program

team and supervisory committee) to be the most appropriate approach to this particular

program’s evaluation. For this reason, the current study proposed a realist evaluation perspective.

Realist evaluation (RE) is a theory-driven measurement and evaluation framework that was

developed in the late 1990s, most notably by Ray Pawson and Nick Tilley (1997). It is praised

for its pragmatism, and most importantly, because of its context-specific approach, RE is

particularly well suited to complex interventions, which are defined by their context (Rogers,

2008). The FNP meets all criteria for a “complex intervention,” including the presence of

multiple, interacting components; non-linear trajectories; multiple feedback mechanisms; and

multiple alternative and simultaneous causal strands, as what works for one family may not work

or may unfold differently for another (Rogers, 2008). Thus, RE was a logical choice that aligned

with the FNP’s own program philosophy, development, daily operation and program goals.

RE is grounded in scientific realism, a school of thought necessitating the inclusion of both

observable and latent aspects of the world in our theories and models because both have real

world effects (Bhaskar, 1979; Leplin, 1981; Miller, 1987; Putnam, 1982). RE and scientific

realism are distinguished from critical realism, which assumes that the inability in social sciences

to create the ‘closed system’ study available to the natural sciences necessitates an approach

centred on abstract a priori reasoning and acknowledging that a moral lens is applied to all

evaluation (Boyd, 1989).

Page 28: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

15

In the context of program evaluation, RE presumes that social programs, such as the FNP, are

intended to address a social problem, such as the struggle for families to quickly find appropriate

MHA care for their youth where and when they need it most (Pawson & Tilley, 1997). RE

surmises that programs generate outcomes by providing resources (e.g. information, skills,

support) and/or influencing their participants’ reasoning (such as their values, beliefs or

attitudes), and thus decision-making. Because of this, the way in which these social programs

‘work’ in any given interaction strongly depends on the participant’s response to the resources

provided by the program (Pawson & Tilley, 1997). This is why RE encourages open-endedness

in knowledge gathering efforts - due to the theoretically infinite number of contingencies that

could continue to shape understanding of the phenomenon or program of study.

The contexts in which a program operates, in terms of multi-level social, economic and political

structures (such as participant subgroups, stakeholders, program staff characteristics, or larger

social, cultural, economic or political conditions) are also extremely important to understanding

how a program achieves change because as little as one factor can influence whether or not a

particular mechanism is triggered for a participant, either by influencing reasoning directly or via

how resources are provided (Pawson & Tilley, 1997).

Again, RE is theory-driven; its hallmark is the distinctive “generation” of causality. As such, a

program theory or a series of conjectures about how an intervention is proposed to work is

needed to guide the evaluation design (Pawson & Tilley, 1997). Realist program evaluations are

intended to answer the question, which goes on to form the program theory, “what works, for

whom, in what respects, to what extent, in what contexts, and how?” The question is addressed

by gathering information on observable and latent mechanisms that explain how any given

outcome was caused - what resources were provided, and what was the participant’s reasoning in

response - and what the influence of the context was:

C + m(resources) + m(reasoning) = O.

In RE, analyses result in a series of conjectures relating contexts (C) and mechanisms (M) to

outcomes (O), known as “C-M-O configurations,” each of which is associated with certain risks

and assumptions. C-M-O configurations are arrived at by hypothesizing and testing individual

dyads of data (i.e. C-C, C-M, C-O, et c.), which are then amalgamated into C-M-O triads. C-M-

O configurations, most notably individual dyads, serve several purposes. The first is to add to the

Page 29: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

16

description of the dataset in terms of the relationships between variables; the second is to

confirm that hypothesized covariates are related to the outcomes of interest; and the third is to

satisfy inferential modeling assumptions. The most robust C-M-O configurations are then often

tested in inferential models and compared to the initial program theory for iterative refinement

and retesting (Pawson & Tilley, 1997).

Because of the importance of context, realist evaluation is method-neutral and encourages a wide

range of approaches to data collection (Dalkin, Greenhalgh, Jones, Cunningham, & Lhussier,

2015). However, most often, quantitative methods are used to evaluate context and outcome,

while qualitative methods are used to further describe these factors and gather in-depth

information on mechanisms, which are typically social or psychological processes that can be

difficult to adequately capture using quantitative methods.

2.1 Theory-building process

Another tenet of RE is the importance of involving the program team in the evaluation design

process (Pawson & Tilley, 1997). In the context of the current study, this included collaborating

to develop a conceptual framework, program theory and corresponding logic model underlying

family navigation, all of which then informed the measurement framework. This approach

aligned well with the FNP’s stakeholder-driven philosophy as families, providers and

policymakers were intentionally involved in project design and development from the early

conceptualization stages; and evaluation is an explicit part of the program model.

The conceptual framework, program theory, logic model and outcomes of interest in the current

study were developed in collaboration with the program team over a series of meetings

facilitated by the PI, beginning in August 2015.

The process began with a discussion of the concept of family navigation and the construction of

an appropriate definition for the family navigation services by using their own program

information as a starting point. This included promotional materials such as educational

pamphlets, website information, and the FNP’s business case (Roberts & Schmidt, 2012). This

information was then compared with existing definitions for navigation and examples of similar

constructs, such as care coordination and case management, from available peer-reviewed and

grey literature. After several iterations, the group came to a consensus on the following baseline

Page 30: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

17

definition of family navigation: “Family Navigation is the provision of continuous needs-specific

and family-centred information and support in order to facilitate timely identification of and

access to appropriate resources for the youth and their family.”

It is worth noting that this definition is consistent with the findings of the AHRQ review on the

topic discussed earlier, which determined that while no consensus on the definition of navigation

exists, a few authors do distinguish navigation by its focus on helping to overcome patient-

perceived barriers to care (McDonald, et al., 2007). The definition used here reiterated the idea

that navigation should be client-based and focused on overcoming perceived barriers, rather than

a predefined set of services, which is how care coordination is defined and measured.

The second step was to collaboratively develop a logic model that represents how the FNP is

thought to work (see Figure 1, Appendix B). The process began with a discussion with the

program team, again facilitated by the PI, of program goals and the needs of the target

population. Then, the inputs and range of activities that were required to meet those needs were

determined, in addition to the results and impacts of those activities (tangible outputs). A series

of tools and techniques were employed by the PI throughout the process. These included

articulation of mental models, which involves questioning key informants about how they

understand an intervention to operate and contribute to addressing the needs of their target

population; group model building, in which a models of system dynamics are built using causal

loops based on perceptions of the problem; Strength Weakness Opportunity Threat (SWOT)

analyses, and hierarchical results chains, which identify the chain of effects from inputs through

activities, outputs and outcomes [Bryson, 2011; Richardson & Andersen, 1995; Vennix, 1996).

This extensive list of activities and outputs was then aligned and condensed into a list of key

defining components detailed in Table 1 (Section 1.2.2).

Next, the logic model (Figure 1, Appendix B) was used to clarify the program theory. The PI

facilitated a discussion with the program team focused on the identification of themes within and

across the range of inputs, activities and outputs associated with the outcomes in the logic model.

The program team was also encouraged by the PI to recall past example narratives that helped

clarify how one component linked to another; and prompted to include how any contextual

factors, and any risks and/or assumptions might enable or hinder any step along the way. The

results of this discussion were recorded by the PI and then integrated with literature that offered

Page 31: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

18

evidence for the stated problem (e.g. lack of resources for youth, particularly transition-aged;

lack of coordination; calls for “navigation”); and evidence for why the possible solutions

generated (e.g. the provision of continuous needs-specific and patient-centred information and

support in order to access appropriate resources) would address the problem.

An overall theory about how the program is believed to work was constructed and presented by

the PI to the program team for further refinement. Based on the theory-building process

described above, the program theory hypothesizes that three key mechanisms – accessibility,

continuity of care, and family involvement - directly enable the three outcomes of interest -

family empowerment, improved family quality of life and family service satisfaction.

2.1.1 Program theory

The preliminary program theory was as follows:

➢ IF we supply the inputs that allow us to employ Navigators who are continuously

accessible, interpersonally skilled and adept at assessing and meeting needs from a

family-oriented perspective;

➢ THEN families will have ongoing access to family-based, needs-specific and supportive

care that best responds and adapts to the family’s changing needs;

➢ SO THAT families will have a better understanding of their needs, the range of options

available to them, and how to quickly and efficiently access sufficient and appropriate

resources to meet their needs;

➢ SO THAT families feel empowered to better manage their situation; and

➢ SO THAT families are able to improve their quality of life; and

➢ SO THAT families will be satisfied with the services they receive.

The last step was for the PI to synthesize findings from the literature and conclusions from group

discussions in order to produce a succinct conceptual framework for family navigation (Table 1,

Appendix B). Together, the logic model for the program and conceptual framework for family

navigation proposed that in the presence of certain contextual factors, three key “mechanisms”

characterize family navigation – family involvement, accessibility, and continuity of care –

which enable the project goals and outcomes of interest of empowering families, improving their

family quality of life, and ensuring service satisfaction. Included concepts are described below.

Page 32: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

19

2.2 Conceptual framework

2.2.1 Context

There are several categories of contextual characteristics with bases in the literature and program

theory that were a) included for descriptive purposes; and b) theorized to influence outcomes and

client-reported experience with the FNP to varying extents. It should be noted that because the

FNP employs a “family model of care” in which the family is the client (based on the premise

that it is families (i.e. parents or other primary caregivers) who facilitate the interaction between

a youth and the service systems), in this study, individual-level characteristics of the youth were

conceptualized as attributes of the family (Chovil, 2009).

Categories of contextual characteristics first included demographics, namely age and gender of

the youth, since MHA concerns are known to manifest differently across both variables. Because

navigation was originally designed to help disadvantaged populations overcome perceived

barriers (and because cultural and financial barriers to MHA care are a frequent occurrence) to

evaluate equity, other demographics of interest included geographical location, ethnicity, and

socioeconomic status. Type of MHA concern(s) was the second main category and included

whether care was being sought for mental health concerns, addiction concerns, or both, since

mental health and addiction services are often located in separate service silos and as such, care

for concurrent disorders can be especially challenging to navigate. It also included the specific

MHA concerns, since there are some concerns for which the FNP may be particularly adept at

navigating – relatively common concerns like depression and anxiety - or which are acute and

particularly responsive to attention – suicidality, for example. Other concerns are theorized to be

inherently more difficult to address and/or have fewer designated resources available; this would

be typical of chronic complex conditions like OCD, personality and eating disorders, for

example. Relatedly, youth’s level of acuity, in terms of whether or not families perceived

improvement since enrolling with the FNP, was expected to influence mechanisms and

outcomes, particularly service satisfaction. It is important to reiterate that reported concerns are

parent-perceived; and parent-perceived MHA concerns cannot be equated with formally

diagnosed mental disorders. However, because this study is centred on family experiences and

outcomes, family-reported MHA concerns were determined to be the most relevant variable.

Page 33: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

20

Also theorized to influence mechanisms and outcomes were current program use characteristics,

including the length of time and intensity with which they were engaged with the FNP and

whether they were actively receiving services or had been discharged at the time of evaluation.

Extent and nature of past service utilization was also a category of interest that was intended to

further describe the sample and included variables such as whether they had a formal diagnosis,

previous emergency department (ED) visits, or inpatient stays. Other past service use variables

were expected to influence mechanisms and/or outcomes, such involvement with the justice

system or Children’s Aid Society were, which can add additional layers of complexity to

navigation cases. Reason for contact was the last contextual category of interest and was

included to contribute toward understanding who is using navigation and for what purposes; and

because it was expected that certain reasons would be more difficult to accommodate than

others. For example, families contacting the FNP in search of general recommendations,

information and/or family support may be more likely to have their needs met than families

contacting the FNP in need of a relatively inaccessible resource like residential treatment.

Altogether, a range of contextual factors were included in the conceptual framework, some of

which were included primarily to add to the description of the dataset, whereas others were

specifically hypothesized to influence mechanisms and/or outcomes for families receiving

services from the FNP.

2.2.2 Mechanisms

2.2.2.1 Accessibility

Accessibility was the first proposed mechanism. In this case, it referred to the accessibility of

expertise (i.e. the Navigators), information, and supports. Data from the program team and

families with lived experience were both firm that a program responsive to families’ needs would

be highly accessible, but there were many different examples of what this entailed. For some,

this meant having extended hours to cover weekends and evenings or having the opportunity to

call, email, Skype or attend in person; for others, it meant having someone make the direct

referral for them; and for others, it meant having a Foundation so families who could not afford a

private treatment option, despite high need, would have that opportunity available to them.

Page 34: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

21

To address the range of perspectives, the current framework employed Roy Penchansky and J.

William Thomas’ (1981) classic model of accessibility, which provides a comprehensive

taxonomy of five distinct components of “access” - availability, accessibility, accommodation,

affordability, and acceptability. Each of these components were determined to be relevant to the

current study. Availability refers to the adequacy of supply of FNP services and resources in

relation to families’ needs. Accessibility refers to the relationship between the location of supply

and the location of families, taking account of factors such as transportation resources and travel

time, distance, physical accessibility and cost. Accommodation refers to the relationships

between how FNP services and resources are organized, such as the appointment and client

record systems, hours of operation, meeting facilities, telephone and web-based services; a

family’s relative ability to accommodate to these factors; and their perception of appropriateness.

Affordability refers to the relationship of cost of services (both the FNP itself and the services to

which it refers) to the clients’ ability to pay; this includes “client perception of worth relative to

total cost.” Finally, acceptability refers to the relationship between a family’s attitudes about

what the personal and practice characteristics should be and what the actual characteristics

actually are.

2.2.2.2 Continuity of care

The second theorized mechanism is continuity of care. The framework adapted Canadian

researcher Jeannie Haggerty and colleagues’ conceptualization, which suggests three types of

continuity should be offered to families: 1) informational continuity, the use and accumulation of

personalized information over time on which providers can draw to ensure current care is

appropriate; 2) management continuity, a consistent, coherent and responsive management of a

care plan that can provide a sense of predictability and security, especially to families who are

facing MHA crises; and 3) relational continuity, arguably the most essential need, which refers to

the ongoing therapeutic relationship between patients/families and provider(s) (Haggerty, et al.,

2003). Recall that in focus groups, as per Dr. Freeman’s analogy, families most strongly

emphasized the need for someone to “get in the boat.”

2.2.2.3 Family involvement

“Family involvement” was the final proposed mechanism. This was agreed to be one of the

defining features of the program and its measurement was intended to capture the client-centred

Page 35: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

22

nature of navigation as it applies to families who facilitate the interaction between youth and

service system, and as such, are the central dimension of the care planning system (Tannenbaum,

2001). Family involvement is increasingly referred to in the youth mental health literature,

although no single, empirical definition is known to exist. Inter-related terms include family

empowerment, family-centredness, family-focused, and family-driven (Chovil, 2009; Curtis &

Singh, 1996). In the current framework, a decision was made by the PI in consultation with the

program team to employ Wood’s definition of family involvement as “respecting families as

experts on their children, enlisting them as partners in the care of their children, supporting them

in their caregiver role, and involving them as partners in decision-making at all levels of the

system” (Wood, 2004, p. 6). This conceptualization was selected because it best emphasized the

need to understand family members as part of the context, mechanism and outcome; and as

experts on their children, making them invaluable partners in care planning (McCammon,

Spencer, & Friesen, 2001).

2.2.3 Outcomes of interest

Three main outcomes of interest were derived in collaboration with the program team following

the facilitated discussion regarding program goals. As a unique grassroots project in its early

stages of operation, the program team’s primary interest was in the subjective experience of their

clients. They acknowledged the difficult-to-treat nature of MHA concerns, and the many

unavoidable barriers to help-seeking and positive outcomes. Furthermore, they understood

themselves to broker treatment services rather than provide them directly. As such, they were

most interested in knowing that the program and the time, money and extensive efforts invested

to be responsive to families’ needs had made some positive difference(s) in the daily lives of the

families they set out to help; and whether families were satisfied with their services or not (and if

not, why, and how they could improve).

In the facilitated discussions, members of the program team mentioned wanting to offer families

“a sense of relief,” to “encourage confidence” and “hope for the future,” and “help families help

themselves.” They specifically wanted families to feel more in control of and better able to

manage their youth and family’s situation because of the accessible, needs-responsive

informational, management and relational support they offered. Accordingly, the three outcomes

chosen for the current study were as follows.

Page 36: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

23

2.2.3.1 Family empowerment

Empowerment is not a particularly well-defined construct in the literature, and yet the term is

widely used across health care settings. Definitions, measures and underlying conceptual

frameworks vary significantly and particularly across conditions but tend to revolve around

patients building the beliefs, knowledge and skills to actively participate in their care and

improve their situation. In mental health settings, efforts to facilitate empowerment are generally

associated with positive perceptions and outcomes and have been encouraged for use with

families of youth with extreme behavioural disorders (Battaglino, 1987; Dunst, Trivette, Davis,

& Cornwell, 1998; Freund, 1993; Kopp, 1989; Singh, 1995; Staples, 1990).

Family empowerment has been previously defined as “a process by which the families access

knowledge, skills and resources that enable them to gain control of their own lives as well as

improve the quality of their lifestyles” (Singh, 1995, p. 13). It requires both changes in conscious

beliefs and attitudes, as well as “practical knowledge, solid information, real competencies,

concrete skills, material resources, genuine opportunities, and tangible results” (Staples, 1990, p.

30). Many of these definitions position empowerment as an interactive process – a mechanism

involving both resources and the family’s reasoning – but empowerment can also be

conceptualized as a state in which “a family perceives itself as being able to successfully

negotiate the […] system,” and “efficiently utilize it to meet their needs” (Curtis & Singh, 1996,

p. 504). However, this state is likely dynamic, changing over time in response to evolving

experiences, conditions and circumstances. This idea is reflected in another often-cited definition

of empowerment as “the ongoing capacity of individuals or groups to act on their own behalf to

achieve a greater measure of control over their lives and destinies (Staples, 1990, p. 30). For this

reason, empowerment can theoretically be conceptualized as both a process and a state, an

individual or collective characteristic, and one that can be facilitated in a wide range of settings

and circumstance (Koren, DeChillo, & Friesen, 1992).

In this study, because family empowerment was an explicitly stated program goal in and of itself,

family empowerment was conceptualized as an outcome of successful navigation. That is, family

empowerment was theorized to be a state produced, at least in part, by the process of navigation.

This approach aligned with emerging literature suggesting empowerment is a valuable patient-

Page 37: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

24

reported outcome independent of health status or service utilization, and particularly relevant in

chronic conditions like MHA concerns (McAllister, Dunn, Payne, Davies, & Todd, 2012).

2.2.3.2 Family quality of life

When a youth has an MHA concern, the situation affects the entire family. Youth exist within an

ecological framework in which family is central; family systems theory defines families as goal-

directed, self-correcting, dynamic, interconnected systems that influence and are influenced by

their environment and inherent qualities (Klein & White, 1996). That is, families have different

strengths capabilities, resiliencies and skills that can significantly impact a youth’s health and

family’s functioning; and each family member is linked to the other such that what impacts one

impacts all. Disabilities are thus challenging for everyone involved in the interconnected system,

and family functioning often declines as a result. Thus, in family models of care, outcomes

reflecting enhanced family management and quality of life are increasingly desired (Chovil,

2009).

Family quality of life (FQOL) is an evolving construct in the literature that is increasingly being

used to offer a more nuanced and robust understanding of families’ daily lives when interacting

with the health care system (Gill & Renwick, 2007; Park, et al., 2003). Impact on FQOL is an

important outcome of quality of service delivery in health services research because it may

reflect the nature, content, accessibility and delivery of services and consequently inform quality

improvement (Turnbull, et al., 2000). However, due to its inherently complex nature, there have

been few and disjointed attempts to measure the concept comprehensively and holistically. The

rationale for the application of the FQOL framework in family support service evaluations is

located in the idea that “families that function well support societies, and families with effective

quality of life are seen as a social resource;” and the best way for disability service systems to

improve FQOL is therefore through family-centred intervention models which assess families’

perceived needs and support and build upon families’ strengths to help them function better as a

system (Isaacs, et al., 2007; Samuel, Rillotta, & Brown, 2012). Families are typically the primary

participant in their youth’s help-seeking and care planning, and are increasingly being

understood as experts on their children who are able to provide meaningful and relevant

information from a unique perspective that can significantly improve the appropriateness of care.

Page 38: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

25

Moreover, families who participate in their youth’s care are more likely to feel in control and as

if their needs were met (Curtis & Singh, 1996; Koren, et al., 1997).

Optimal FQOL is conceptualized as having a) perceived needs met, and b) appropriate

opportunities to make active choices according to need (Renwick, Brown, & Raphael, 1998).

There are several domains of FQOL the FNP expected to impact, including family interaction,

parenting, wellbeing, and available support. Navigators provide information and cultivate

knowledge and awareness, skills and strategies, resiliency and capacity to cope, which was

theorized to result in a better understanding of their role and ability to manage daily activities

and interactions with both the youth and the service system. The supportive environment of the

FNP itself, long-term involvement by the navigators, and connections to additional support

networks can increase resilience, support coping skills, decrease perceived burden and help to

reduce the stigma, shame and embarrassment that often accompany help-seeking for MHA

concerns. This, alongside increased access to appropriate and relevant disability-related supports

that meet the perceived needs of both the youth and other family members, are hypothesized to

reduce caregiver burden and stress and improve both physical and emotional well-being.

For the FNP, the family is the client and the key resource for change in the youth’s health and

family’s functioning. As such, family-focused outcomes were preferred by the program team,

who again wanted to know whether families were “better off in some way,” in their day-to-day

life because of the services offered.

2.2.3.3 Service satisfaction

Satisfaction is one of the most widely used patient-reported health care service outcomes that has

value in and of itself; as a key indicator of quality of care; and by examining its predictors, in

informing quality improvement initiatives (Al-Abri & Al-Balushi, 2014; Barr, et al., 2006;

Mpinga & Chastonay, 2011). Measures can reflect both client expectations and the actuality of

services provided (Ware, Synder, Wright, & Davies, 1983). Although cancer-specific, what few

evaluations exist do suggest that patient navigation is associated with improved self-reported

service satisfaction over treatment as usual (Campbell, et al., 2010; Hook, Ware, Siler, &

Packard, 2012; Seek & Hogle, 2007).

Page 39: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

26

Families’ satisfaction with the navigation services received was an essential indicator of success

for the FNP, a program specifically designed to respond to families’ stated needs. In 2014, the

FNP methodically developed an in-house satisfaction measure to evaluate both satisfaction with

the navigation services, and satisfaction with the services to which Navigators referred families

were of interest in the current study. It was suggested that this satisfaction measure be employed

in the current study both because it is already validated and because data resulting from the

current study could then be compared to data from the pilot study of that measure.

Page 40: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

27

Chapter 3 Research Questions and Rationale

Research questions

With the conceptual framework and program theory fleshed out, three broad research questions

were determined to guide this study:

1. a) Who is the Family Navigation Project serving?

b) Is the Family Navigation Project reaching its target population?

c) Overall, are families satisfied with the services they received?

2. a) Do families perceive the Family Navigation Project to be providing accessible,

continuous, family-inclusive care?

b) How does context influence perceived experience of the program?

3. a) Do families who perceive the Family Navigation Project as accessible,

continuous and family-inclusive experience better outcomes in terms of family

empowerment, family quality of life, and service satisfaction?

b) How does context influence these outcomes?

3.1 Hypotheses

With regard to the third research questions, which asks whether a family’s context and perceived

experience of navigation (defined by accessibility, continuity and family involvement) influence

the outcomes of interest, several broad hypotheses following from the program theory were

proposed and are outlined in Table 2 below.

Table 2. Hypotheses by research question and dependent variable

Research question Dependent

variable Hypothesis

Do families who perceive

the Family Navigation

Project as accessible,

Family

empowerment

H1: Context influences outcomes to varying

extents and family navigation positively

predicts family empowerment scores

Page 41: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

28

continuous and family-

inclusive experience

better outcomes in terms

of family empowerment,

family quality of life, and

service satisfaction?

How does context

influence these

outcomes?

Family quality

of life

H1: Context influences outcomes to varying

extents and family navigation positively

predicts family quality of life

Navigation

satisfaction

H1: Context influences outcomes to varying

extents and family navigation positively

predicts navigation satisfaction

Satisfaction

with referred

services (SRS)

H1: Context influences outcomes to varying

extents and family navigation positively

predicts satisfaction with referred services

3.2 Objectives

This study was further guided by several broad objectives. The first was to develop a conceptual

framework and testable program theory for family navigation. The groundwork for this objective

was laid through extensive collaboration by the PI with the program team to design the current

study and the underlying conceptual and measurement framework. The hope was that the

resulting data supported this framework, which could then be iteratively retested and refined in

future evaluations.

The second objective was to evaluate perceived experience of family navigation in terms of

accessibility, continuity and family involvement; to report on the influence of perceived

experience on desired outcomes of family empowerment, family quality of life, and service

satisfaction; and to explore the influence of context on experience and outcomes.

Addressing these two objectives would contribute evidence regarding if and how the FNP is

achieving its goals (i.e. having the impact on families that it intended to have); and the various

ways in which a family’s experience may vary across contexts. Should the analyses show

positive associations between program components and the desired outcomes, it would lend

support to the value of family navigation.

The final objective was to test a theory about which mechanisms are underlying the observed

outcomes. If mechanisms were found to have a significant statistical impact on the outcomes of

interest, this would help to answer the questions of if and how family navigation helps families,

and what the relative influence of context was for families in this sample. Identifying the

underlying mechanisms is particularly important to implementation of the model elsewhere.

Page 42: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

29

Although implementation elsewhere would need to take local context into account (different

populations and/or settings may have different needs), the underlying mechanisms should remain

the same.

3.3 Rationale for the current study

The FNP navigates for all family members. That this evaluation was limited to family-based

outcomes was a conscious choice that reflects this, as well as the critical understanding that it is

families who primarily facilitate the interaction with the health care and social services systems

when seeking care for their youth. The decision was also practical; only a very small number of

FNP clients are youth seeking care for themselves and are unlikely to engage in outcomes

research, leading to a small and inconsistent sample.

The study, as designed, was warranted for several reasons. First, ongoing evaluation and change

in response to feedback was a key tenet of this evolving program, and this study provided an

opportunity to test an evaluation framework for the concept of family navigation. Second, since

the FNP was intended to be a prototype for future navigation initiatives and was expected to

scale up, it was especially important to collect data that could be used by program developers to

inform its implementation and success in other jurisdictions. Lastly, this study has the potential

to make a significant contribution to the literature base. Navigation is increasingly being

employed across a range of health care settings despite the lack of literature, particularly around

conceptual and evaluation frameworks, and the lack of distinction from care coordination, case

management and related terms. This study yields both a conceptual framework for family

navigation and a mixed methods measurement framework, as well as a series of testable program

theories with both qualitative and quantitative data to the literature base.

Page 43: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

30

Chapter 4 Measurement

Specification of variables

4.1 Context variables

The context variables explicitly included in the conceptual framework were collected as part of

the FNP’s standard intake procedure and as such, could be abstracted from the client charts.

Recall from Section 2.3.1 that contextual characteristics of interest fell into several categories.

The first main category of contextual variables was demographics, including age, gender, and

geographical location. There were two demographic variables that could not be abstracted from

client charts, however; the FNP currently does not explicitly ask families to self-report their

ethnicity or socioeconomic status. For the purposes of this study, a decision was made by the PI

in consultation with the program team and supervisory committee to add original questions to the

survey package in order to collect these variables of interest.

Another key category of contextual variables was program use characteristics, including the

length of time and intensity (number of contacts) with which they were engaged with the FNP.

Also of chief importance was type(s) of MHA concern(s), both in terms of whether care was

sought for mental health concerns, addiction concerns, or both (and for which specific

conditions); and youth’s level of acuity, in terms of whether or not families perceived

improvement in their youth’s functional status since enrolling with the FNP. Lastly, past service

utilization, including past ED visits, inpatient stays, and involvement in their case by other

service sectors; and reasons for contact were contextual categories of interest given they can add

additional layers of complexity to navigation.

Measurement of context variables is summarized in Table 1, Appendix C.

4.2 Mechanism variables

The three mechanisms proposed by the program theory were operationalized using original

survey questions grounded in peer-reviewed literature. Questions were written by the PI, in

consultation with the supervisory committee, and validated for face validity by the program team

and pilot sample. Accessibility was measured using five five-point ordinal scale items that

Page 44: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

31

reflected each of Penchansky and Thomas’s (1981) accessibility framework – availability,

accessibility, accommodation, affordability, and acceptability; refer back to Chapter 3.3.2 for

further detail on each of the mechanisms. Similarly, continuity of care was measured using three

original five-point ordinal scale items that reflected the three types of continuity of care proposed

in Haggerty and colleagues’ (2003) continuity of care framework – informational continuity,

management continuity, and relational continuity. Lastly, a single five-point item was written to

measure family involvement (Wood, 2004). Five-point ordinal scales ranged from “very

dissatisfied (1)” to “very satisfied (5).”

The details of each mechanism measure, including the proposed survey questions, are

summarized in Table 2, Appendix C.

4.3 Outcome variables

Measurement of outcome variables is summarized in Table 3, Appendix C, and is discussed in-

depth by outcome below.

4.3.1 Family empowerment

Family empowerment was quantitatively operationalized using the Family Empowerment Scale

(FES), a scale with which the program team was already familiar. Developed in the 1990s by

P.E. Koren, N. DeChillo and B.J. Friesen for use with families who have children with emotional

and developmental disabilities, this measurement tool since been extensively used in a variety of

related settings (1992). The scale includes three separate subscales representing family

empowerment at different levels or spheres of influence: at home in the family (FES Family), in

seeking services (FES Service-seeking), and advocating in the community (FES Community).

The first dimension, level of empowerment, can occur at the family level, in terms of their ability

to manage their immediate circumstances at home; the service system level, in terms of families’

ability to actively command and negotiate the services necessary for their youth and family’s

needs; and at the community/political level, in terms of families’ ability to advocate for

improved services for the youth MHA population in general. The second dimension measures the

expression of empowerment at any of the above levels through families’ attitudes; knowledge

and skills; and behaviour.

Page 45: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

32

The FES consists of 34 items, each containing a subjective statement (for example, “I believe I

can solve problems with my child when they happen”) and a five-point response scale ranging

from “not true at all (1)” to “very true (5).” The scoring strategy reflects the three levels of

empowerment by summing scores on the items from each level (family, service system and

community/political) to yield three separate subscores. The scale has previously been extensively

psychometrically evaluated and shows strong internal consistency (Chronbach’s alpha ranges

from 0.87 to 0.88 for each of the three levels) and good test-retest reliability (Pearson’s

correlations ranging from 0.77 to 0.85, and paired t tests for mean differences between subscores

were non-significant). With regard to validity, inter-rater agreement is also good, with kappa

coefficients ranging from 0.70 to 0.83 across levels and 0.77 overall. Lastly, factor analysis

suggests the items generally correspond to the levels conceptualized in the tool.

4.3.2 Family quality of life

Family quality of life (FQOL) was quantitatively operationalized using a modified version of the

Beach Center Family Quality of Life Scale (BCFQoLS) (Hoffman, Marquis, Poston, Summers,

& Turnbull, 2006). The tool was developed on the basis of two empirical studies conducted by

the Beach Center on Disability at the University of Kansas, in collaboration with families,

service providers and researchers. The goal for the researchers was to create an instrument that a)

accurately reflected families’ perception of the most important aspects of FQOL, b) was

psychometrically sound for use in research studies, and c) would be a pragmatic tool for use in

program evaluation and policy planning. Like the FES, the BCFQoLS was originally designed

for use with families who have youth up to age 21 with disabilities, but can be adapted to the

youth MHA setting.

The BCFQoLS consists of 20 items scored on a five-point response scale ranging from “very

dissatisfied (1)” to “very satisfied (5).” The items span five conceptual domains of family life:

family interaction, parenting, emotional well-being, physical/material well-being, and disability-

related support. Scores can be summed by domain and/or in total for a continuous measure.

Psychometric analyses, including confirmatory factor analyses with a sample of 488 families,

support the five-factor model and suggest that all subscales individually have good to excellent

fit for both importance and satisfaction rating (Chronbach’s alpha was 0.94 and 0.88,

respectively). Subscales correlate significantly with hypothesized similar measures, suggesting

Page 46: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

33

convergent validity; and test-retest reliability correlations were significant at the 0.01 level

across the board.

Following consultation with the program team and supervisory committee, the PI removed

several items in this scale due to irrelevancy to the population and modified wording to increase

relevancy. In addition, feedback from the program team led the PI to add three more items

intended to capture the full range of expected impacts of family navigation of FQOL. The first

added item asks families about their sense of control over care. This was deemed highly relevant

by the PI because a) it was a stated goal to improve families’ ability to manage their daily lives,

and b) as mentioned earlier, existing evidence suggests that families who participate in their

youth’s care are more likely to feel in control and as if their needs were met (Curtis & Singh,

1996). The second added item asks families about their hopefulness and future outlook. When

reviewing potential outcome measures, this concept of hopefulness and future outlook was found

to be a unique and desirable feature of the Bakas Caregiving Outcomes Scale (Bakas, 2014). As

such, a corresponding item was added by the PI that similarly captures the FNP’s stated goal of

providing families with hope for the future of their youth. Lastly, an item subjectively assessing

overall FQOL was added. What little FQOL literature does exist has shown a degree of discord

between a total sum of the rating of individual items (which is how the BCFQoLS is scored), and

the rating of a single overall satisfaction item rating of FQOL on a five-point Likert scale; this

finding aligns with the well-known tendency for satisfaction measures to be negatively skewed

(Samuel, et al., 2012). A comparison of ratings on this single item versus the summed total score

allows for the contribution of further evidence toward this topic.

Following the removal and addition of specific items described above, the resulting modified

scale (“m-BCFQoLS scale”) remained at 20 items, which were summed to a single continuous

variable representing a total cross-sectional score.

4.3.3 Service satisfaction

Satisfaction with navigation services was operationalized using the FNP’s in-house satisfaction

survey tool mentioned earlier (Section 2.3.3). The “NAVSAT” was developed by the FNP in the

summer of 2014 in response to a gap in the literature around valid and reliable measures specific

to navigation in the field of MHA (Fishman & Levitt, 2014). No similar MHA navigation

satisfaction scales are known exist. The scale was developed by the FNP team, which included

Page 47: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

34

staff navigators, a medical director, project manager, and parent advisory council members at

Sunnybrook Health Sciences Centre. Questions were adapted from six valid and reliable

satisfaction scales in the literature, including the Verona Satisfaction Scale and the UKU-

Consumer Satisfaction Rating Scale. This 25-item scale consists of two sections: satisfaction

with navigation, and satisfaction with the resources to which families were referred by the

navigation team. Items are rated on five- or seven-point Likert-style scales.

The first section includes 15 items that ask families to assess their satisfaction with the treatment

recommendations received, their navigator’s knowledge and fluency in the mental health and

addictions systems, respect for confidentiality, and nature/frequency of contact. Designated

outcome variables include likelihood of recommending the service, navigator helpfulness, and

overall service satisfaction. For the purposes of the current study, scores on the three outcome

variables were summed to a continuous total score representing satisfaction with navigation

services. The second section presents 10 items that evaluate families’ satisfaction with the

referred resource in terms of type, delivery method, location and effectiveness. The designated

outcome variable for this section was a seven-point item measuring overall satisfaction with

referred services. For the purposes of this study, this item was treated as a continuous variable,

an approach that is considered generally acceptable for variables with more than five categories

(Rhemtulla, Brosseau-Liard, & Savalei, 2012).

The NAVSAT was previously validated in a sample of 80 families receiving services from the

FNP in July 2014. The results suggested the scale was easy to administer, took 10-15 minutes to

complete, and had excellent psychometric properties that reflect the scale’s grounded

development process. The first 15-item section on navigation satisfaction yielded a Chronbach’s

alpha value of 0.957, and in factor analyses, all variables loaded on a single factor, with loadings

ranging from 0.776 to 0.936. The second 10-item section on referred service satisfaction

similarly yielded a reliability coefficient of 0.948, with a factor analysis and scree plot

suggesting a one-factor solution was again most appropriate (factor loadings in this section

ranged from 0.702 to 0.913) (Fishman & Levitt, 2014).

4.4 Qualitative measures

To enhance the robustness of the data, participants were prompted to qualitatively expand on

their context, experiences and outcomes at multiple points throughout the survey. The survey

Page 48: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

35

was organized such that quantitative context, mechanism, and outcome measures were presented

on their own pages. As a means of efficiently collecting concurrent qualitative data that would

help to enhance understanding of the quantitative responses, following the completion of each

page or measure, participants were asked: “Do you have anything you would like to add?”

(Driscoll, Appiah-Yeboah, Salib, & Rupert, 2007). Responses could then be entered in unlimited,

open-text fields. As with all questions in this survey, qualitative responses were entirely

voluntary.

4.5 Measurement validity

All measures were extensively refined and validated for face validity, clarity, comprehension,

applicability, and relevancy in collaboration with the program team, who had an in-depth

understanding of the nature of the study, having extensively participated in the design of the

conceptual framework and program theory. However, it was also necessary to verify whether

families understood and responded to the questions and underlying concepts as expected. This

type of validity evidence, based on response processes, was first proposed by S. Messick and

refers to adequacy with which respondents’ actions and thought processes reflect their

understanding of the construct in the same way as intended by the researchers, such that the

rationale for use of the measure and interpretation is maintained (1990).

To further validate the measures, open-ended prompts were embedded in the survey package to

gather feedback from a randomly selected pilot sample of families from the FNP registry (n=30,

or to theoretical saturation). The preamble explained that as the first group of participants, they

were being asked to provide feedback intended to improve the survey experience for the formal

study phase. Open-ended prompts related to a) the convenience and ease of use of the survey

platform and b) the understanding of the survey items for the remaining participants. Feedback

generated from the pilot sample was thematically analyzed, amalgamated with feedback from the

program team, and proposed refinements re-presented to the program team for final consensus.

The pilot sample data were not included in the study sample for the analyses.

Page 49: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

36

Page 50: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

37

Chapter 5 Methods

Design overview

This multi-phase, mixed methods study employed a cross-sectional design whereby both

quantitative and qualitative data were collected via an online survey package including both

open- and closed-ended items. Data subsets were then examined individually and in combination

via statistical and thematic analyses with findings merged to produce a more comprehensive

understanding of whether and how family navigation is helping families.

5.1 Study setting

This study was conducted electronically from September through December 2016 in Toronto,

Ontario.

5.2 Ethics

The study is considered to be minimal risk so only delegated and administrative reviews were

required. Ethics approval was jointly sought from and granted by Sunnybrook Research Institute

(SRI) and University of Toronto using the Toronto Academic Health Sciences Network

(TAHSN) Human Subjects Research Ethics Application.

5.3 Sample

The intended sample for the study was any individual officially registered with the FNP at the

time of the study who was seeking care on behalf of a youth with an MHA concern. This

represented approximately 95% of the client roster at the time of the study (n=1045) and

excluded youth seeking care for themselves due to the family-focused nature of the outcomes

employed. Families who did not provide the FNP intake coordinator with verbal consent to begin

navigation services or who did not provide an email address (or who indicated they did not to

want be contacted by email) were excluded (n=303). All individuals meeting inclusion criteria

were identified and abstracted from the client roster by the program’s intake coordinator and

research fellow, resulting in an eligible convenience sample of 742 who were registered with the

FNP at the time of the study.

Page 51: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

38

Since the FNP is a relatively new program whose reach is continuously extending throughout the

GTA, the client roster has grown considerably since the study took place. At the time of the

study, the sample of 742 individuals represented approximately 65.0% of the roster.

5.4 Data collection

5.4.1 Survey methods

Data collection occurred over two four-week phases beginning in September 2016; a pilot phase

preceded the official launch of the survey. All individuals identified by the FNP as meeting the

established inclusion criteria were surveyed using a modified Dillman’s Tailored Design Method

(Dillman, et al., 2014). Communication regarding participation in the study was initiated by an

FNP staff member personally known to the clients. The FNP team suggested that communication

by email is generally their clients’ preferred method of contact, and that web-based surveys were

most likely to yield the highest response rate, so eligible individuals were sent an introductory

email briefly describing the study, along with a request to participate. Interested individuals were

instructed to follow a live link to access a personal letter from the PI more fully explaining the

nature and goals of the study, and an option to proceed to the web-based survey platform.

At the time of administration, the survey platform, FluidSurveys, was a Canadian company

which stored data on Canadian servers only; all data was exported prior to FluidSurveys’

acquirement by American counterpart, SurveyMonkey. Upon following the live link, participants

were presented with an online consent form detailing the full protocol, all details of participation

including risks, benefits, ethical concerns and time to complete, as well as an assurance that all

data would be deidentified and confidential. Only individuals who provided electronically signed

and dated consent were permitted to continue to the survey questions; failure to provide consent

redirected individuals to the exit page. Individuals were also asked for consent to be contacted

directly by the PI should any follow up questions about the nature of their responses or

opportunities for participation in follow up research arise.

Individuals who did not respond to the introductory email received a reminder email with a

replacement survey package following one and three weeks’ non-response, respectively. Surveys

were accepted up to four weeks after the initial email, a cut-off point that was selected based on

the program team’s experience that families who did not respond within that time period were

Page 52: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

39

unlikely to respond at all. Indeed, only a couple surveys were submitted after the four-week

mark. Lastly, as compensation for their time and effort, families who did complete surveys were

offered the option to have their name entered in a draw for a $150 Loblaws gift certificate.

5.4.1.1 Pilot phase overview

Of the 742 eligible individuals, 30 were randomly selected by the intake coordinator to receive

an invitation to participate in a pilot phase of the survey in September 2016. The purpose of the

pilot phase was to gain any client perspectives that would allow further refinement of face and

content validity of the survey items beyond what the program team was able to contribute. The

survey was administered as per the protocol and respondents were prompted in open fields for

feedback on convenience of the survey platform and understanding of the survey items for each

scale, along with any generalized suggestions for improvement. Feedback suggested the survey

was clear, asked important questions, and was not overly long but it required some modifications

including simplifying wording on one item, correcting a few formatting errors, and addressing

some technical difficulties with the website. All feedback and suggestions were reviewed by the

PI in consultation with the program team and the supervisory committee before being

implemented into the final version of the survey accordingly.

5.4.1.2 Response rates

5.4.1.2.1 Pilot survey phase

Of the 30 randomly selected individuals invited to participate in the pilot phase of the survey,

one email was found to be invalid, 11 individuals responded by following the live link to the

survey package, and five of those 11 respondents completed the informed consent and provided a

complete set of responses. The complete response rate (16.7%, or five of 30) was lower than

expected. To better understand and address this low rate, the PI presented the results to the

program team, who helped to identify several potential explanatory variables that informed the

PI’s subsequent decision not to modify the recruitment process.

The first explanatory variable discussed was logistical and related to the fact that the pilot study

was initiated on the first day of school in a population whose youth are school-aged and

struggling with mental health concerns that are often related to school attendance and experience.

This is a busy, stressful and inconvenient time for parents and youth alike where voluntary, time-

Page 53: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

40

consuming surveys are unlikely to take priority. The second related variable discussed was that

there is extensive variability in the nature, intensity and severity of MHA concerns families that

come to the FNP are dealing with, and some families were likely too entrenched in their

difficulties to be in a position to participate, even if they had wanted to. However, this is not a

feature that was easily identified from the client roster. Lastly, it was noted that the roster

consisted of individuals who had registered with the FNP at any point since the program

launched in 2014, and it is possible that some individuals were less motivated to provide

feedback on an experience that may have taken place as long as two years ago.

With regard to the low completion rate, the survey platform allows users to identify at which

point in the survey respondents exit. It was noted that three of the six participants who failed to

complete the survey did not proceed beyond the introductory letter from the PI, and so a decision

was made to shorten the introductory letter, simplify the study description further, and highlight

the minimal time required to participate in hopes of engaging more individuals in the formal

survey phase. A contingency plan to have the FNP’s Medical Director send out the email was

arranged. The survey pages themselves were also shortened and further simplified, and a

progress bar was included at the bottom of each page.

5.4.1.2.2 Formal survey phase

The survey was formally launched following the integration of pilot phase feedback in

November 2016. The individuals invited to participate in the pilot phase were removed from the

list of potential participants, resulting in an eligible sample of 712; and of 712 email invitations

sent out, 24 addresses were found to be invalid and bounced back. The resulting valid study

sample was 688.

Of 688 individuals who were sent an introductory email, 276 (or 40.1%) followed the live link to

the survey package, which is slightly more but comparable to the pilot phase response rate. Of

these 276 individuals who accessed the survey package, 48.6% (n=134) provided electronically

signed informed consent and a complete set of responses, which again is slightly more but

comparable to the pilot phase completion rate. Of the 688 eligible individuals, 134 complete

responses yield an overall response rate of 19.5%.

Page 54: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

41

Participants were encouraged to reach out to the PI with any questions or concerns about the

survey or their participation. Several individuals highlighted an unfortunate technical difficulty

with the website whereby individuals repeatedly lost their progress when they attempted to take a

break and return to the survey at a later point in time. It is likely that this frustrating experience,

an unexpected feature of the survey platform and beyond the control of the investigator,

discouraged completion to some extent.

5.4.2 Chart review

Client chart reviews were completed using a secure, cloud-based management software called

EMHware, which is an advanced electronic medical record platform primarily intended for use

by outpatient mental health care agencies in Ontario. As per the measurement framework, a

number of additional contextual variables were abstracted from client charts for all survey

respondents who provided both their full name and informed consent to access personal health

information (n=134). Chart reviews took place over several weeks following the survey, in

December 2016.

The purpose of the chart review was to collect additional information about the nature of a

family’s concerns and engagement with the FNP, while reducing additional burden on the client.

For example, logistical details like clients’ status (active or discharged), documented dates of

contact (to calculate time engaged with the program), address (required to identify catchment

area), and history of previous service use were abstracted from charts. Also abstracted were

reported MHA concerns, and more nuanced details like school avoidance, history of bullying,

and any legal involvement. A complete list of variables by source is provided in Appendix C.

5.5 Sample size and power

Although the resulting sample size was smaller than anticipated, the Rule of Ten for multiple

regressions with more than six predictors, along with Green’s comprehensive guide indicating

that N > 50+8m where m is the number of predictors is sufficient for multiple correlations, and N

> 104 + m is adequate for testing individual predictors, all suggest that the current study’s

sample size of 134 is sufficiently powered to accommodate as many as 10 predictors (Green,

1991; VanVoorhis & Morgan, 2007). Further, statistical analyses were supplemented with

qualitative data, the goal of which was to gain a more robust understanding of any patterns that

Page 55: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

42

emerged versus relying entirely on standardized but decontextualized statistical indicators of

model fit and power.

5.6 Data storage

Only the PI has access to identifying information. All data was deidentified, and a separate file

linking clients to their unique identifiers is kept in a locked drawer in a locked office on a

password-protected and encrypted hard drive. Data will be stored for the recommended 10 years

as per hospital policy, and then destroyed.

5.7 Overview of analytical approach

A multi-phase, mixed methods analytical plan was developed to evaluate the data according to

the theoretical framework, program theory and research questions. The first phase involved a

quantitative descriptive analysis of the variables in order to assess the distribution of the

variables and describe the sample with regard to who was using the FNP at the time of the study,

for what reasons, what their outcome scores were, and how they were distributed.

The second phase involved a quantitative correlational analysis, the purposes of which were to

test hypothesized relationships within and between select context, mechanism and outcome

variables, generating a series of C-M-O configurations that further described the sample and

supported the program theory (refer back to Chapter 2 for an overview of C-M-O

configurations). Results of this second phase were then used to expand upon the descriptive

analysis; and to inform the selection of covariates and refine hypotheses for inferential modeling

in a third, quantitative phase. This final inferential quantitative phase was implemented to

statistically test the impact of the proposed context and mechanism variables on the outcomes of

interest using a series of generalized linear models.

A criticism of Realist Evaluation has been the lack of clarity around how to construct C-M-O

configurations, as few realist evaluations clearly delineating C-M-O construction have been

published. The analytical approach in the current study was thus primarily informed by the

methods and resulting data types, but also draws from and expands upon one of the few

examples in the literature in which a method for incrementally building C-M-O configurations is

well-described (Byng, et al. 2005).

Page 56: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

43

Finally, a qualitative phase followed the quantitative analyses in order to enhance the robustness

of the analysis. As mentioned earlier, open-text fields prompting qualitative feedback were

included throughout the survey package. This qualitative data was then described, thematically

coded and analyzed in relation to key themes from the program theory. The analytical approach

is further detailed at the beginning of each phase in the chapters that follow.

Page 57: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

44

Chapter 6 Quantitative Analysis

Overview of quantitative approach

RE is a theory-driven framework and as such, the quantitative analysis followed a deliberate,

theory-building process designed to accumulate as much information as possible about the

relationships between context, mechanism and outcome variables. First, descriptive statistics

were used to determine for whom the FNP is navigating and for what reasons; how clients rated

the services they received; and how they scored on the outcomes of interest. Descriptive statistics

were also used to assess the nature of the data in order to ensure that the most appropriate

statistical procedures were chosen.

Following descriptive analyses, correlational analyses were used to identify factors that tend to

co-occur. Correlational analyses in this study served several foundational purposes. Correlations

amongst context variables themselves (C-C dyads) helped to further describe the population

seeking navigation, including the complex relationships within and between demographics,

mental health and addiction concerns, and interactions with the service system. Correlations

within and between mechanisms and outcomes were also employed to ensure preliminary

modeling assumptions were met; M-M dyads reflected risk of multicolinearity (subsequently

informing and resulting in the use of a single component score), and M-O dyads indicated

whether the assumption that the mechanism is linearly related to the outcome had been met.

However, the primary goal of the correlational analyses was to identify meaningfully correlated

C-M and C-O dyads, which were used as covariates in outcome models.

6.1 Importing and coding of data

All survey responses and client chart data were exported from the FluidSurveys and EMHware

platforms, respectively, and imported into a pre-organized data collection form in Microsoft’s

Excel where data were combined, coded and scored as per the measurement framework and

variable specifications. Once all survey and chart data was amalgamated for each individual, data

was deidentified using unique client identity numbers. All variables were then imported, defined

and analyzed using IBM’s SPSS Statistics 22.0 (IBM Corporation, 2013).

Page 58: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

45

6.2 Data quality

Overall data quality was excellent. Data for all descriptive variables were complete for all

individuals with the exception of missing data (n=12) for self-reported household income. Data

for all mechanism variables (accessibility, continuity of care, and family involvement) were also

complete for all individuals. With regard to outcome data, single item responses on the family

empowerment scales were missing entirely at random in a small handful of cases (n=5) and so

the decision was made to impute the middle value in these instances (equivalent to the response,

“Neither”). However, there was systematically missing data that resulted from cases in which

respondents completed the survey in full through the family empowerment scales, but then

completed only some or none of the remaining outcome measures (n=5). In these cases, data

were coded as missing and excluded pairwise from analyses to avoid overfitting (Tabachnick &

Fidell, 2007). In addition, several outliers were identified but based on the theoretical

framework, were retained in the analyses as they are thought to reflect the inherent variability in

families’ experiences. In this framework, every case is understood to have a unique ability to

contribute to understanding who navigation works best for, when and why.

The significance of variables was interpreted using an alpha level of 0.05 where appropriate.

6.3 Descriptive analyses

Descriptive analyses were performed on all variables. The particular statistical approach

employed was determined by the type of variable.

6.3.1 Context variables

Descriptive statistics including frequencies and measures of central tendency, where appropriate,

are presented for each category of context variables (refer back to Section 2.3.1 and 4.1):

demographics, MHA concern characteristics, current program and past service use

characteristics, and reasons for contacting the FNP. This section is intended to answer the first of

the guiding research questions, which was “Who is the Family Navigation Project serving? Is the

Family Navigation Project reaching its target population?”

Page 59: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

46

6.3.1.1 Demographics

6.3.1.1.1 Clients

The vast majority of the sample, 93.3%, are parents who were seeking help for their own youth

and family, although other client relationships such as step-parents and grandparents were

reported. Since the roster included clients registered from 2014 onward, only 27.6% were

actively receiving services at the time of the study, while 72.4% were classified as discharged.

Tests of group differences indicated that discharge status at the time of the survey was not

meaningfully related to any variables of interest. However, due to the highly variable and often

chronic nature of MHA concerns, clients may be discharged and reactivated multiple times

according to the needs of the family at any point in time. The number of re-engagements was not

measured in the present study but may have been associated with the total number of weeks a

family spent engaged with the program. The average length of time families spent as a client of

the program varied significantly across the sample, ranging from as little as one week to 106

weeks (or just over 2 years), with clients speaking to their Navigator anywhere from three to 100

times. Clients were actively engaged for an average of 24 weeks (SD=21.7; median=17.1), or six

months, and had an average of 19.7 documented contacts with a Navigator (SD=14.9; median

16.0). As indicated by the range and standard deviation, both length of time and number of

contacts varied widely across the sample.

The FNP operates independently from but is still affiliated with SHSC. Correspondingly, clients

were located across the City of Toronto and the GTA; 29.1% of clients lived in SHSC’s official

catchment area, 41.8% live in the City of Toronto outside the SHSC catchment, and 28.4% live

in the GTA. One client in Nova Scotia was included.

As a whole, the sample consisted primarily of high income families, with 65.6% reporting

household incomes over $100,000 and 40.2% alone over $150,000. This sample is also 85.1%

Caucasian, which does not reflect demographic MHA trends in the highly diverse Toronto area

(Ontario Human Rights Commission, 2015).

6.3.1.1.2 Youth of the clients

The demographic trends among the youth in this sample reflected demographic trends in the

youth MHA literature (ICES, 2017; Kessler., et al., 2005, 2012; Statistics Canada, 2006). The

Page 60: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

47

majority of youth in this sample were males (59.0%) who lived at home with their family

(72.4%). The FNP specifies an age range of 13 to 26 years but occasionally accepts youth under

or over the range on a case by case basis. A large majority of youth in this sample are older teens

and young adults; transitional-aged youth (18 to 24 years old) in particular account for 61.9% of

the sample. Youth ranged in age from 12 to 25 and over. Descriptive statistics are presented for

continuous and categorical contextual variables in Table 3 and Table 4, respectively, below.

Table 3. Descriptive statistics for continuous current program use variables

Variable

(n=134) Mean Median SD Min Max IQR Skew Kurtosis K-S

Weeks engaged 24.0 17.2 21.7 1 106.1 24.1 1.6* 2.5* 0.165**

No. contacts

with Navigator 19.7 16.0 14.9 3 100 14 2.3* 7.7* 0.204**

*Statistic is more than twice its standard error **p<.01

Table 4. Frequency statistics for demographic variables

Variable N % of cases*

(N=134)

Demographics

Youth age

12 to 15 years 9 6.7%

16 to 17 years 23 17.2%

18 to 19 years 28 19.9%

20 to 24 years 55 41.0%

25 years and over 19 14.2%

Youth gender

Male 79 59.0%

Female 55 41.0%

Youth ethnicity

Caucasian 114 85.1%

Asian 7 5.2%

Other 10 7.5%

Family household income $25 to 49,999 6 4.9%

Page 61: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

48

(n=122) $50 to 74,999 14 11.5%

$75 to 99,999 20 16.4%

$100 to 149,999 31 25.4%

$150k+ 49 40.2%

Family location

Sunnybrook catchment 39 29.1%

City of Toronto 56 41.8%

GTA 38 28.4%

Family living arrangement

with youth

Home with family 97 72.4%

On their own 18 13.4%

Treatment centre 8 6.0%

Home with single parent 7 5.2%

Family member’s relation to

youth

Parent 125 93.9%

Other 9 6.7%

*Percentages and totals may not add to 100.0 due to rounding and/or because cell counts < 5 are suppressed

6.3.1.2 Mental health and addiction characteristics

Clients reported a wide range of youth MHA concerns. Nearly the entire sample (98.5%)

reported at least one mental health concern (mean = 2.3), the most common of which was

depression (57.5% of cases), followed by anxiety (48.5%). ADD/ADHD (22.4%), suicidality

(defined as suicide attempts or ideation; 17.9%), OCD (14.9%), and self-harm (12.7%) were the

next most frequently reported concerns. Less common but still significant were chronic, severe,

and difficult-to-treat concerns including personality disorders, eating disorders, and bipolar

disorder. A history of bullying was reported in 18.7% of cases; and 27.6% of cases explicitly

reported school avoidance or refusal. Descriptive statistics for mental health and addiction

characteristics are presented for continuous and categorical variables in Tables 5, 6 and 7,

respectively.

In this sample, 45.5% of clients (n=61) reported concurrent mental health and addiction

concerns. Refer to Table 7 for the full range of reported addiction concerns in this sub-sample,

Page 62: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

49

most of which were assumed to co-occur with a mental health concern as only two cases reported

standalone addiction concerns. Clients who reported addiction concerns reported an average of

1.7 concerns. The most common concern was cannabis use, reported in 74.6% of cases with

addiction concerns (35.1% of total cases). Alcohol use was also a frequent concern, reported in

44.4% of addiction cases (20.9% of the total sample). Among addiction concerns, the next

highest reported rate was for stimulant use (17.5%), which includes cocaine, ecstasy, MDMA,

and methamphetamines. It is also important to note that these MHA concerns were parent-

reported and did not make the distinction between use and abuse or dependence. Other reported

substance use concerns in this sample included opioids, benzodiazepines and non-opioid

prescription drugs, and behavioural addictions, such as to the Internet, video games, and sex.

Nearly two thirds of clients reported that since enrolling with the FNP, their youth’s functional

status had stayed the same (64.2%), while just over a quarter had improved (26.9%) and a

handful had worsened (9.0%).

Table 5. Descriptive statistics for continuous mental health and addiction characteristic

variables

Variable N Mean Median SD Min Max Skew Kurtosis K-S

Number of mental

health concerns 134 2.3 2.0 1.4 0 7 0.7* 0.4 0.165**

Number of addiction

concerns 63 1.7 1.0 1.1 0 5 1.5* 2.0* 0.295**

*Statistic is more than twice its standard error **p<.01

Table 6. Frequency statistics for mental health characteristic variables

Variable N % of cases*

(N=134)

Youth functional status since

enrolling

Improved 36 26.9%

Stayed the same 86 64.2%

Worsened 12 9.0%

Reported mental health Depression 77 57.5%

Anxiety 65 48.5%

Page 63: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

50

concerns** ADD/ADHD 30 22.4%

Suicidality 24 17.9%

OCD 20 14.9%

Self-harm 17 12.7%

Personality disorder 13 9.7%

Eating disorder 10 7.5%

Developmental disorder 9 6.7%

ODD/Anger 9 6.7%

Bipolar disorder 8 6.0%

Panic attacks 7 5.2%

Psychosis 6 4.5%

Trauma 5 3.7%

Bullying Yes 37 27.6%

No 97 72.4%

School avoidance Yes 37 27.6%

No 97 72.4%

Concurrent mental health

and addiction concerns

Yes 61 45.5%

No 73 54.5%

*Percentages may not add to 100.0 due to rounding and/or because cell counts < 5 were suppressed **Indicates multiple response variable

Table 7. Frequency statistics for addiction characteristic variables

Variable N % of cases

(N=134)*

% of addiction

cases (N=63)

Reported addiction

concerns**

Cannabis 47 35.1% 74.6%

Alcohol 28 20.9% 44.4%

Stimulants 11 8.2% 17.5%

Opioids 5 3.7% 7.9%

Page 64: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

51

Internet and gaming 5 3.7% 7.9%

*Percentages may not add to 100.0 due to rounding and/or because cell counts < 5 were suppressed **Indicates multiple response variable

6.3.1.3 Previous service use

Previous service use characteristics gave insight into the nature and intensity of families’ service

needs. Nearly all families contacting the FNP (92.5%) had already received prior MHA services

for their youth. Despite the numerous and wide-ranging mental health and addiction concerns

reported by parents, only approximately 60.0% of youth had received a formal diagnosis from a

physician or allied mental health professional at some point in their help-seeking journey.

Further, 38.1% of youth had had at least one previous ED visit, and 40.3% had had at least one

inpatient stay.

A case history of legal involvement (which includes involvement by the Youth Justice system

and Children’s Aid Society) was theorized to impact help-seeking by adding another service

sector to navigate. Prior or current legal involvement was reported in over a quarter of cases

(26.9%).

Previous service use characteristics are presented in Table 8 below.

Table 8. Frequency statistics for service use characteristic variables

Variable N % of cases*

(N=134)

Service use

characteristics

Previous service use

Yes 124 92.5%

No 10 7.5%

Current formal diagnosis

Yes 80 59.7%

No 54 40.3%

Previous ED visit(s)

Yes 51 38.1%

No 83 61.9%

Previous inpatient stay(s) Yes 54 40.3%

No 80 59.7%

Page 65: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

52

History of legal

involvement

Yes 36 26.9%

No 98 73.1%

*Percentages may not add to 100.0 due to rounding and/or because cell counts < 5 were suppressed

**Indicates multiple response variable

6.3.1.4 Reasons for contact

The FNP collects information from its clients upon intake with regard to their reasons for

contacting the program. This data was abstracted from client charts during the chart review

phase, and the results indicate that a wide range of reasons was reported (Table 9).

Most often (in 70.1% of cases), clients contacted the FNP actively seeking recommendations for

their given scenario. The next most common reason clients connected with the FNP was to seek

out family support (38.8% of cases). Many families were also simply seeking information

(34.3% of cases).

The majority of the remaining reported reasons for seeking navigation were more specific and

deliberate, reflecting the wide range of needs for families who have a youth with MHA concerns.

A quarter of clients were specifically seeking a psychiatrist, and another 19.4% were specifically

seeking a therapist. A number of clients were also seeking residential treatment (15.7% of cases).

Less commonly, clients contacted the FNP seeking addiction-specific programs.

A range of other reasons for contacting the FNP were reported, including need for crisis

supports, day treatment, aftercare, supportive housing, case management, out-of-province or

country treatment, psychoeducational assessments, and cognitive behavioural therapy (CBT).

The type and frequency of reasons for contact is presented for in Table 8 below.

Table 9. Frequency statistics for reasons for contact variable

Variable N % of cases*

(N=134)

Reason for contact**

Recommendations 94 70.1%

Family support 52 38.8%

Information 46 34.3%

Page 66: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

53

Psychiatrist 33 24.6%

Therapist 26 19.4%

Residential treatment 21 15.7%

Addiction program 12 9.0%

Crisis supports 6 4.5%

Day treatment 5 3.7%

*Percentages may not add to 100.0 due to rounding and/or because cell counts < 5 were suppressed

*Indicates multiple response variable

6.3.2 Mechanism variables

This section addresses the first half of the second guiding research question, “Is the Family

Navigation Project providing accessible, continuous, family inclusive care? Which contextual

factors matter?” Descriptive statistics related to the frequency distribution of each of the three

mechanism variables are reported below. The program theory guiding this study suggested that

family navigation (as per the definition developed during the theory-building stage of this study)

is characterized by three key mechanisms: accessibility, continuity of care, and family

involvement. The evaluation feedback will inform understanding of whether clients perceived

the services they received as highly accessible, continuous, and family-inclusive.

6.3.2.1 Accessibility

Accessibility is a widely known and accepted literature-based concept that was assessed using

five items each rated on five-point Likert-style scale; the sum of the item ratings was calculated

to produce a continuous total score out of 25. Overall, accessibility was rated very highly across

the sample (mean = 21.1, median = 23, range = 5 to 25, SD = 4.9). There are a handful of outliers

reflecting a handful of clients with low ratings, which widened the standard deviation.

Examining the individual items, the ratings were highly comparable for the first four (mean =

4.3, median = 5.0), but on the last item, which represented appropriateness of care and asked

whether the FNP met their expectation of what family navigation entailed, ratings were slightly

lower (mean = 3.9, median = 4.0).

Page 67: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

54

Normality was assessed using a range of descriptive statistics. A median to the right of the mean;

a non-linear Q-Q plot; a boxplot with high positive values, a long tail and several outliers;

negative skew and positive kurtosis values twice their standard error; and a significant

Kolmogorov-Smirnov (K-S) statistic (0.241, p=.000) all led to the conclusion that the

distribution of total accessibility scores were significantly skewed and leptokurtic.

Table 10 presents the statistics for total score and for each item, respectively.

Table 10. Descriptive statistics by item for the “Accessibility scale”

Variable: Accessibility (5 items; n=134)

Item Mean Median SD Skew Kurtosis K-S

To what extent do you feel you were

able to reach the Navigator whenever

you needed? (/5)

4.32 5.0 0.91 - - -

To what extent do you feel it was

convenient to communicate or meet with

the Navigator? (/5)

4.25 5.0 1.18 - - -

To what extent do you feel the Navigator

accommodated your schedule when

making arrangements with you? (/5)

4.37 5.0 1.1 - - -

To what extent do you feel the Navigator

was considerate of your resources? (/5) 4.29 5.0 1.12

- - -

To what extent do you feel the Navigator

met your service expectations for family

navigation? (/5)

3.90 4.0 1.31 - - -

Accessibility total score (/25) 21.13 23.0 4.90 -1.7* 2.3* 0.241**

Scale: 1=Very dissatisfied, 2=Dissatisfied, 3=Neither, 4=Satisfied, 5=Very satisfied

*Statistic is more than twice its standard error **p<.001

6.3.2.2 Continuity of care

Continuity of care, like accessibility, is a literature-based concept that was assessed using three

items each rated on five-point Likert-style scale; the sum of the item ratings provided a

continuous total score out of 15. Overall, continuity of care was also rated very highly (mean =

Page 68: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

55

11.8, median = 13.0, SD = 3.8) and the distribution followed a similar pattern to accessibility. A

histogram, median to the right of the mean, non-linear Q-Q plot, boxplot with high positive

values and a long tail, negative skew value twice its standard error, and significant K-S statistic

(0.218, p=.000) indicated that the distribution of scores was also significantly negatively skewed.

However, relative to accessibility, continuity ratings are lower. This is because there was a

greater proportion of low-scoring outliers on this measure and so the mean is relatively lower

and the standard deviation relatively wider. Inter-item means were also slightly lower for one

item in particular, which asked clients whether their Navigator “continuously communicated and

coordinated” with the family and other service providers.

Table 11 presents the statistics for total score and for each item, respectively.

Table 11. Descriptive statistics by item for the “Continuity of care scale”

Variable: Continuity of care (3 items; n=134)

Item Mean Median SD Skew Kurtosis K-S

To what extent do you feel the Navigator

continuously communicated and

coordinated information with you and the

service providers to whom you were

referred? (/5)

3.88 4.0 1.35 - - -

To what extent do you feel the Navigator

continuously adequately responded to

changes in your family’s situation and

needs? (/5)

3.90 4.0 1.40 - - -

To what extent do you feel the Navigator

was continuously committed to

understanding and helping your family

until you no longer felt you require their

services? (/5)

4.01 5.0 1.35 - - -

Continuity of care total score (/15) 11.80 13.0 3.83 -1.1* 0.1 0.218**

Scale: 1=Very dissatisfied, 2=Dissatisfied, 3=Neither, 4=Satisfied, 5=Very satisfied

*Statistic is more than twice its standard error **p<.001

Page 69: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

56

6.3.2.3 Family involvement

Unlike the accessibility and continuity of care measures, the family involvement measure in this

framework was a single five-point ordinal scale item that was based in the literature developed

by the PI in consultation with the program team and supervisory committee. Like accessibility

and continuity of care though, family involvement followed similar score patterns and was rated

very highly by the sample overall (mean = 4.0, median = 4.0, mode = 5.0, SD = 1.3). Again, all

plots and statistics suggested the distribution was similarly significantly negatively skewed (K-S

= 0.284, p=.000), but as with the accessibility and continuity of care measures, there were also

some low-scoring clients.

Table 12 present the descriptive statistics for this item.

Table 12. Descriptive statistics for the single item “Family involvement”

Variable: Family involvement (1 item; n=134)

Item Mean Median SD Skew Kurtosis K-S

To what extent do you feel the

Navigator consistently involved you

and your family in all stages of care

planning and decision making?

(/5)

4.0 5.0 1.3 -1.2* 0.2 0.284**

Scale: 1=Very dissatisfied, 2=Dissatisfied, 3=Neither, 4=Satisfied, 5=Very satisfied

*Statistic is more than twice its standard error **p<.001

6.3.3 Outcome variables

The following section reports descriptive statistics for each of the outcomes of interest. In

building the program theory, the program team was asked to identify its goals, which were

empowering families, improving their quality of life, service satisfaction (Figure 1, Appendix B).

Several representative quantitative measures were then identified and slightly modified to better

fit the population: the Family Empowerment Scale (FES), a modified Beach Center Family

Quality of Life Scale (-mBCFQoLS), and the NAVSAT tool. Descriptive statistics for NAVSAT

responses in particular answer research question 1c: “Overall, are clients satisfied with the

services they receive?”

Page 70: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

57

6.3.3.1 Family empowerment

The FES consisted of 34 items across three separate subscales representing family empowerment

at different levels or spheres of influence: at home in the family (FES Family), in seeking

services (FES Service-seeking), and advocating in the community (FES Community). The

continuous total subscale scores could be used individually, or summed to create a total

empowerment score. Only the first two subscales (FES Family and FES Service-seeking) were

selected for use in this study as the community-level scale was later determined to be too

conceptually distal a measure to expect it would be meaningfully or statistically impacted by the

program mechanisms proposed in this study; and that including community subscale scores as a

third of a “total empowerment score” would limit ability to detect a statistical impact at the more

relevant and proximal family and service-seeking levels. A decision was thus made by the PI in

consultation with the program team and supervisory committee not to proceed with scores from

the community-level subscale, and to use scores for the two subscales (FES Family and FES

Service-seeking) as individual outcome measures rather than summing to a single family

empowerment score.

6.3.3.1.1 FES Family

The family subscale of the FES (FES Family) was measured using 12 five-point items summed

to a continuous total score out of 60. Unlike the mechanism scores, empowerment scores were

more evenly distributed. The mean for the sample was 43.9 (median = 44, SD = 6.9, IQR = 8).

All normality statistics were non-significant, and plots confirmed that scores on this variable may

be slightly leptokurtic but can safely be assumed to follow a normal distribution with long but

even tails.

Table 13 presents the statistics for item-specific and total scores.

Table 13. Descriptive statistics by item for the “FES Family subscale”

Variable: FES Family subscale (12 items; n=134)

Item Mean Median SD Skew Kurtosis K-S

When problems arise with my child, I handle

them pretty well (/5) 3.65 4.0 0.73

- - -

Page 71: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

58

I feel confident in my ability to help my

child grow and develop (/5) 3.49 3.0 0.87

- - -

I know what to do when problems arise with

my child (/5) 3.34 3.0 0.76

- - -

I feel my family life is under control (/5) 3.37 3.0 1.0 - - -

I am able to get information to help me

better understand my child (/5) 3.49 4.0 0.96

- - -

I believe I can solve problems with my child

when they happen (/5) 3.14 3.0 0.80

- - -

When I need help with problems in my

family, I am able to ask for help from others

(/5)

3.80 4.0 1.07 - - -

I make efforts to learn new ways to help my

child grow and develop (/5) 4.25 4.0 0.77

- - -

When dealing with my child, I focus on the

good things as well as the problems (/5) 4.05 4.0 0.78

- - -

When faced with a problem involving my

child, I decide what to do and then do it (/5) 3.77 4.0 0.85

- - -

I have a good understanding of my child’s

disorder or behavioural concern (/5) 3.57 4.0 0.94

- - -

I feel I am a good parent (/5) 3.93 4.0 0.77 - - -

FES Family subscale total score (/60) 43.84 44.0 6.29 -0.6 2.4 0.074

Scale: 1=Very dissatisfied, 2=Dissatisfied, 3=Neither, 4=Satisfied, 5=Very satisfied

6.3.3.1.2 FES Service-seeking

Like the family subscale, the services subscale (FES Service-seeking) included another 12 five-

point items, summed to a total score out of 60. Scores on the child services subscale were similar

to the family subscale scores, with equivalent means and medians (43.9 and 44.0, respectively)

Page 72: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

59

but slightly more variability across the sample (SD = 9.0, IQR = 12). This was reflected in a

negative skew statistic more than twice its standard error, a Q-Q plot revealing a slight

curvilinear pattern, and a boxplot with relatively less even tails due to a greater number of low-

scoring outliers. However, the K-S normality statistic was non-significant and so the distribution

was cautiously specified as normal.

Table 14 presents the statistics for total score and for each item.

Table 14. Descriptive statistics by item for the “FES Service-seeking subscale”

Variable: FES Service-seeking subscale (12 items; n=134)

Item Mean Median SD Skew Kurtosis K-S

I feel that I have a right to approve all

services my child receives (/5) 3.57 4.0 1.25

- - -

I know the steps to take when I am

concerned my child is receiving poor

services (/5)

3.30 3.0 1.16 - - -

I make sure that professionals understand

my opinions about what services my child

needs (/5)

3.77 4.0 1.07 - - -

I am able to make good decisions about

what my child needs (/5) 3.54 4.0 1.01

- - -

I am able to work with agencies and

professionals to decide what services my

child needs (/5)

3.72 4.0 1.10 - - -

I make sure I stay in regular contact with

professionals who are providing services to

my child (/5)

3.82 4.0 1.18 - - -

My opinion is just as important as

professionals’ opinions in deciding what

services my child needs (/5)

3.85 4.0 1.02 - - -

I tell professionals what I think about

services being provided to my child (/5) 3.55 4.0 1.08

- - -

I know what services my child needs (/5) 3.18 3.0 0.87 - - -

Page 73: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

60

When necessary, I take the initiative in

looking for services (/5) 4.45 5.0 0.75

- - -

I have a good understanding of the service

systems my child is involved in (/5) 3.42 4.0 1.09

- - -

Professionals should ask me what services

I want for my child (/5) 3.78 4.0 1.13

- - -

FES Service-seeking subscale total score

(/60) 43.72 44.0 8.56 -0.8* 1.0 0.075

Scale: 1=Very dissatisfied, 2=Dissatisfied, 3=Neither, 4=Satisfied, 5=Very satisfied

*Statistic is more than twice its standard error

6.3.3.2 Family quality of life

FQOL was scored using a modified version of the BCFQoLS (“m-BCFQoLS scale”) that

consisted of 20 five-point ordinal items summed to a total FQOL score out of 100. Overall,

FQOL scores were similar to family empowerment subscale scores with a median of 4.0 and a

mean in the 70th percentile (mean=72.73, SD=14.3). However, three specific items stood out as

rated lower than the rest, with medians of 3.0 and means less than 3.3; these items related to

availability and accessibility of resources to support families to manage youth MHA concerns.

The first item that was rated lower referred to families’ availability of outside help to take care of

needs; the second item referred to the quality of families’ relationships with service providers;

and the third referred to families’ overall feeling of control over their youth’s MHA care.

Descriptive statistics and normality plots indicate that this variable is normally distributed.

Table 15 presents the descriptive statistics for item-specific and total scores.

Table 15. Descriptive statistics for the “m-BCFQoLS scale”

Variable: m-BCFQoLS scale (20 items; n=131)

Item Mean Median SD Skew Kurtosis K-S

Your family enjoys spending time together

(/5) 3.76 4.0 0.88

- - -

Page 74: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

61

Your family has the supports you need to

relieve stress (/5) 3.43 4.0 1.03

- - -

Your family helps each other out with

schoolwork, tasks or activities (/5) 3.79 4.0 0.86

- - -

Your family talks openly with each other

(/5) 3.66 4.0 0.95

- - -

Your family members get along with each

other (/5) 3.69 4.0 1.04

- - -

Your family members have time to pursue

their own interests (/5) 3.98 4.0 0.92

- - -

Your family solves problems together (/5) 3.50 4.0 0.93 - - -

Your family members support each other

to accomplish goals (/5) 3.79 4.0 0.89

- - -

Your family members show that they love

and care for each other (/5) 4.15 4.0 0.91

- - -

Your family has outside help available to

take care of the special needs of all family

members (/5)

3.37 3.0 1.14 - - -

Adults in your family feel they are able to

parent the youth in your family (/5) 3.50 4.0 1.02

- - -

Your family gets medical or other care

when needed (/5) 4.10 4.0 0.94

- - -

Your family is able to handle life’s ups

and downs (/5) 3.71 4.0 0.90

- - -

Adults in your family have time to take

care of the individual needs of every youth

(/5)

3.73 4.0 0.98 - - -

Your family’s youth with a mental health

and/or addiction problem has support to

accomplish goals at school and/or in the

workplace (/5)

3.51 4.0 1.20 - - -

Page 75: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

62

Your family’s youth with a mental health

and/or addiction problem has support to

make friends (/5)

3.87 4.0 0.95 - - -

Your family has good relationships with

those who provide services and supports

to your family members (/5)

3.25 3.0 1.20 - - -

Your family feels in control over the care

of your youth with a mental health and/or

addiction problem (/5)

2.97 3.0 1.25 - - -

Your family feels more hopeful for the

future of your youth with a mental health

and/or addiction problem (/5)

3.37 4.0 1.22 - - -

Your family feels your overall quality of

life is better (/5) 3.57 4.0 1.07

- - -

m-BCFQoLS total score (/100) 72.73 73.0 14.30 -0.3 -0.6 0.064

Scale: 1=Very dissatisfied, 2=Dissatisfied, 3=Neither, 4=Satisfied, 5=Very satisfied

6.3.3.3 Service satisfaction

As discussed earlier (Section 4.3.3), the NAVSAT is the FNP’s in-house satisfaction measure

that captures both proximal satisfaction with the navigation services received, and more distal

but still relevant satisfaction with the services to which they were navigated. One of the research

questions identified by the program team was, “Overall, are clients of the Family Navigation

Project satisfied with the services they receive?”

6.3.3.3.1 Satisfaction with navigation services

The “NAVSAT total score” was a continuous total score out of 19 that reflected satisfaction with

the navigation services received based on the sum of three ordinal outcome variables: two seven-

point items and one five-point item representing likelihood of recommendation, general

Navigator helpfulness, and overall service satisfaction. The NAVSAT is highly specific to the

program and underwent extensive validation. As such, the three items included in the total score

were simply summed to preserve the integrity of the items as they were originally written, rather

than calculating a weighted total score. What matters in the current study, particularly in terms of

Page 76: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

63

modeling the impact of mechanisms on satisfaction, is the relative difference in total score

between individuals.

Overall, total scores were very high (mean = 15.9, median = 18.0, SD = 3.9). Normality statistics

indicated the distribution is not normal (K-S = 0.238, p=.000); like the mechanisms, it was

significantly negatively skewed and leptokurtic. This was confirmed by a boxplot with high

values and a long tail, and a curvilinear pattern on the Q-Q plot similar to those observed with

the mechanism variables. This was expected given both the sample’s high mechanism scores and

the general tendency for satisfaction measures to yield positive ratings.

Table 16 presents the statistics for item-specific and total scores.

Table 16. Descriptive statistics for the “NAVSAT total score scale”

Variable: NAVSAT total score scale (3 items; n=130)

Item Mean Median SD Skew Kurtosis K-S

In general, how helpful did you find your

Navigator? (/7) 5.76 6.0 1.49 - - -

In general, how satisfied are you with the

Family Navigation Project? (/7) 5.80 7.0 1.57 - - -

How likely are you to recommend this

service to family and friends? (/5) 4.35 5.0 1.01 - -

-

NAVSAT total score (/19) 15.90 18.0 3.9 -1.4* 1.2* 0.238**

Overall, how satisfied are you with the

services to which you were referred?

(/7)

5.0 5.0 1.7 -0.6* -0.2 0.188**

5-point scale: 1=Very unlikely, 2=Unlikely, 3=Not sure, 4=Likely, 5=Very likely

7-point scale: 1=Extremely dissatisfied or unhelpful, 2=Dissatisfied or unhelpful, 3=Fairly dissatisfied or unhelpful, 4=Neither, 5=Fairly satisfied or helpful, 6=Satisfied or helpful, 7=Extremely satisfied or helpful

*Statistic is more than twice its standard error **p<.001

6.3.3.3.2 Satisfaction with referred services

The second outcome of interest derived from the NAVSAT tool was an outcome variable for

satisfaction with referred services. This was a single seven-point item with a mean and median of

Page 77: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

64

5.0 (SD = 1.7), which was also not normally distributed (K-S = 0.188, p=.000). Like the

NAVSAT total score and satisfaction measures in general, the distribution of scores was highly

negatively skewed. However, there was considerably more variability and a significantly greater

proportion scoring the middle value (which equates to “Neither satisfied nor dissatisfied”),

reflected in the fact that this distribution was not at all leptokurtic like the NAVSAT total score

was. This was also evident when examining the boxplot, which showed a much wider spread of

scores with a short upper tail and a very long lower tail. This was not unexpected, since the FNP

can directly manage the ways in which they provide navigation services, but cannot directly

influence the services to which they refer.

Statistics for this item are included in Table 16 above.

The descriptive analysis of the satisfaction measures in particular answered the research

questions of whether, overall, families were satisfied with the services they received. The results

suggested that families in this sample were highly satisfied with both the navigation services

received, and the services to which they were referred. Later in the analysis, the influences of

context and family experience on all outcomes of interest is evaluated.

6.4 Correlational analysis

6.4.1 Overview of approach

As discussed earlier (see Sections 2 and 5.7), RE is a theory-driven framework operationalized

through the use of C-M-O configurations, which are essentially buildable, testable hypotheses

about the relationships between context, mechanism and outcome variables (Byng, et al., 2005;

Pawson & Tilley, 1997). To arrive at C-M-O configurations, select dyads (i.e. C-M, C-O, M-M,

and M-O) were tested from the dataset according to hypotheses generated from the conceptual

framework and program theory (see Chapter 2 and Appendix B). Testing select C-M and C-O

dyads was a process primarily used to provide further rationale for the hypothesized covariates to

be included in the inferential models of each outcome variable, namely by confirming that the

proposed variables were linearly related to the mechanisms and outcomes of interest. Secondly,

M-M and M-O dyads were tested to provide evidence used to satisfy initial inferential modeling

assumptions.

Page 78: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

65

The following section is organized according to the two functions outlined above. Spearman’s

rho was used for ordinal and skewed continuous variables, and the Phi coefficient was calculated

for correlations between two dichotomous variables. To increase statistical reliability, no

variables with cell counts less than 10 were included in correlational analyses. For multiple

response variables, including types of MHA concerns and reasons for contact, numerous

categories were excluded due to small sample size. The significance of correlations was

interpreted at an alpha level of 0.05.

The dataset consisted of many variables that are generally highly interactive. With the number

and type of interrelated variables tested and a relatively modest sample size, correlation

coefficients were expected to be modest in effect size. It is also possible that type 1 error was

inflated. However, a decision was made by the PI in consultation with a statistician to forgo

adjustment for multiple tests in order not to risk underestimating the importance of contextual

factors that are known to be significant for families’ help-seeking experiences and outcomes.

This is a decision that is validated in the literature, which suggests such adjustments (Bonferroni

and false discovery rates, for example) are often far too conservative for real world application,

particularly in social sciences (Armstrong, 2014; Cabin & Mitchel, 2000; Curtin & Schulz, 1998;

Sirotich & Durbin, 2014).

6.4.2 C-M and C-O dyads: Selection of covariates for inferential modelling

The first function of the correlational analysis was to confirm whether or not the hypothesized

covariates for inferential modeling were, in fact, linearly related to the outcomes of interest. This

process also helps to answer the research questions related to which contextual factors matter for

each of the proposed mechanism and outcomes. As discussed earlier (Sections 2.3.1 and 4.1),

select context variables that typically would be expected to influence outcomes were tested for

their relationships with both mechanisms and outcomes.

Hypothesized contextual variables of influence included age, since different service systems

serve different age groups; acuity or youth improvement (i.e. clients whose youth improved were

expected to report better outcomes); concurrent addiction, since mental health and addiction

services are provided by separate sectors; type(s) of MHA concern(s), since some conditions are

inherently more difficult to treat, and since different conditions have different resources

available; and reason for contact, specifically whether clients were seeking a psychiatrist referral

Page 79: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

66

(expected to be positively related to outcomes, given the FNP’s ability to facilitate referrals and

individually match youth with particular treatment providers) or residential treatment, which was

expected to negatively influence outcomes due to the widely acknowledge lack of available

residential services. Other reasons for contact that reflected service gaps like day treatment and

aftercare did not have sample sizes larger enough (>10) to be included in inferential analyses.

Lastly, legal involvement in cases was expected to negatively influence both mechanisms and

outcomes due to the complexity associated with navigating justice system requirements.

The FNP was designed to provide families with needs-based help in overcoming perceived

contextual barriers to care by offering them highly accessible, continuous navigation that

specifically prioritized family involvement. If it is true that these three service mechanisms –

accessibility, continuity of care and family involvement – are what enable the FNP to help

families overcome perceived barriers, then correlational analyses should reveal largely negligible

relationships between clients’ contexts and their mechanism scores. The anticipated exception

was related to the extensiveness of current program (FNP) use. For example, clients who were

relatively more engaged with the program, as determined by the number of documented contacts

with a Navigator, were expected to inherently experience greater accessibility, continuity of care,

and family involvement throughout the process.

The results of the correlational analysis of select C-M and C-O dyads are presented in Table 17,

which follows. With regard to C-M dyads, results were generally as expected. Very few

contextual variables were significantly associated with clients’ mechanism scores; the same three

context variables were found to be significantly associated with all three mechanism variables.

First, as expected, the number of contacts with a Navigator was significantly positively

associated with accessibility (r=.24, p=.005), continuity of care (r=.24, p=.005) and family

involvement scores (r=.21, p=.016). The other two influential context variables were related to

type of MHA concern. Interestingly, suicidality concerns were significantly positively associated

with clients’ scores on all three mechanisms (accessibility: r=.27, p=.001; continuity of care:

r=.20, p=.023; and family involvement: r=.24, p=.005). In contrast, OCD concerns in particular

were negatively associated with clients’ scores on all three mechanisms (accessibility: r=-.29,

p=.001; continuity of care: r=-.21, p=.016; and family involvement: r=-.23, p=.007). Legal

involvement was negatively related with each of the mechanisms but the relationship did not

approach significance. In general, with the exception of suicidality and OCD, mechanism scores

Page 80: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

67

were not significantly associated with context variables that typically influence delivery of

mental health and addiction care.

Compared to C-M dyads, relationships between contexts and outcomes (C-O dyads) were more

readily identified but not to the extent expected (i.e. 100.0% of hypothesized relationships).

While most variables trended in the expected direction, only approximately 30.0% of the

hypothesized C-O dyads were found to be statistically significant. With regard to the two FES

subscales, the same three contextual variables were associated with both; in this sample, older

youth age was associated with lower empowerment scores on both FES Family and FES Service-

seeking subscales (r=-.24, p=.006; and r=-.27, p=.002, respectively). Concurrent addiction

concerns were also associated with lower scores on both empowerment subscales (FES F: r=-.20,

p=.021; FES S: r=-.26, p=.00d). Again, of note, suicidality concerns were positively associated

with clients’ empowerment scores (FES F: r=.21, p=.000; FES S: r=.33, p=.000). Legal

involvement was again negatively, but not significantly, related with empowerment subscale

scores.

Similar to family empowerment, concurrent addiction concerns were also associated with lower

cross-sectional m-BCFQoLS scores (r=-.26, p=.003). Personality disorder concerns were also

negatively associated with m-BCFQoLS scores (r=-.18, p=.036). In contrast, depression concerns

were positively associated with m-BCFQoLS scores (r=.20, p=.025). Consistent with the other

suicidality-related dyads noted above, reports of suicidality concerns were positively associated

with clients’ m-BCFQoLS scores (r=.27, p=.002). In the case of FQOL, legal involvement was

significantly negatively associated with m-BCFQoLS scores (r=-.28, p=.001).

Page 81: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

68

Accessibility total score

Continuity of care total

score

Family involvement

score

FES Family

total score

FES Service-seeking

m-BCFQoLS total score

NAVSAT total score

SRS score

Number of contacts with Navigator

.244** .241** .208** -.016 .085 .100 .229** .104

Age -.050 -.010 -.058 -.235** -.268** -.056 -.016 -.019

Perceived youth improvement

-.115 -.167 -.116 -.017 -.056 -.037 -.146 -.205*

Concurrent addiction -.030 -.026 -.106 -.199* -.255** -.260** -.157 -.127

Type of mental health concern

Depression .132 .160 .113 .076 .045 .196* .186* .195*

Anxiety -.025 -.059 -.095 -.129 -.082 -.128 -.007 .027

ADD/ADHD .087 .033 .046 -.135 -.045 -.144 -.003 -.041

Suicidality .274** .196** .241** .307** .327** .272** .287** .273**

OCD -.289** -.207** -.234** -.055 .109 -.156 -.247** -.254**

Eating disorder

-.011 .054 .059 -.082 -.115 .018 .086 .100

Personality disorder

-.108 -.166 -.113 -.045 .007 -.184* -.121 -.113

Reason for contact

Psychiatrist referral

.140 .133 .112 .035 .033 .055 .198* 199*

Residential treatment

.039 -.006 -.047 -.141 .066 -.137 -.128 -.150

Legal involvement -.040 -.145 -.142 -.138 -.029 -.283** -.227** -.180*

*p<.05 **p<.01

Table 17. Results of a correlational analysis between select C-M and C-O dyads

Page 82: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

69

C- dyads related to satisfaction outcomes followed similar patterns. Satisfaction with navigation

services (NAVSAT total score) was, as expected, positively and significantly associated with

number of contacts with a Navigator (r=.23, p=.009). Satisfaction with navigation was also

positively associated with reported depression and suicidality concerns (r=.19, p=.034; and

r=.29, p=.001, respectively). In contrast, OCD was negatively associated with navigation

satisfaction (r=-.25, p=.005). Whereas concurrent addiction concerns were significantly

negatively associated with empowerment and FQOL outcomes, the negative association between

concurrent addiction concerns and navigation satisfaction did not reach significance (r=-.16,

p=.074). Surprisingly, family-perceived youth improvement was not significantly correlated with

navigation satisfaction. In fact, the data suggested a counter-intuitive inverse relationship

between improvement and satisfaction (r=-.15, p=.098); this should be interpreted in the context

of the fact that the majority of the sample (64.2%) indicated their youth’s status had stayed the

same, with only 9.0% (or 12 families) indicating their youth’s status had worsened.

As expected, the need for a psychiatrist referral was positively associated with navigation

satisfaction (r=.20, p=.024), whereas there was a trend for the need for residential treatment to be

negatively associated with navigation satisfaction (r=-.13, p=.148). Legal involvement was, as

anticipated, significantly negatively associated with navigation satisfaction as well (r=-.23,

p=.009).

Satisfaction with referred services were generally associated with the same contextual variables

as satisfaction with navigation services at comparable effect sizes and levels of significance.

Satisfaction with referred services was positively associated with depression (r=.20, p=.027) and

suicidality concerns (r=.27, p=.002), whereas scores were negatively associated with OCD

concerns (r=-.25, p=.004). Clients seeking a psychiatrist referral had positively associated

satisfaction with referred service scores (r=.20, p=.024). Again, counter-intuitively, perceived

youth improvement was found to counter-intuitively be negatively, and this time significantly,

associated with satisfaction with referred services scores (r=-.21, p=.020). Lastly, legal

involvement was also significantly negatively associated with satisfaction with referred services

(r=-.18, p=.041).

Based on the correlation results of C-M and C-O dyads, predictors and covariates that were

confirmed to be significantly linearly related to the mechanisms and outcomes were selected for

Page 83: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

70

inclusion in subsequent inferential models and are summarized according to outcome of interest

in Table 18 below. The exceptions were that a decision was made by the PI to preserve

concurrent concerns in the model of navigation satisfaction both because it trended near

significance, but more importantly, because it had a strong theoretical rationale for inclusion.

Similarly, youth improvement was maintained in the model of satisfaction with referred services

despite the counter-intuitive significant inverse relationship between improvement and

satisfaction observed.

Table 18. Summary of covariates and predictors for modeling by outcome

Outcome

domain

and

variables

Family empowerment Family

quality of life Satisfaction

FES F FES S m-

BCFQoLS NAVSAT SRS

Mechanisms

Accessibility

Continuity

Family

involvement

Accessibility

Continuity

Family

involvement

Accessibility

Continuity

Family

involvement

Accessibility

Continuity

Family

involvement

Accessibility

Continuity

Family

involvement

Contextual

covariates

No. contacts

with Navigator

No. contacts

with Navigator

No. contacts

with Navigator

No. contacts

with Navigator

No. contacts

with Navigator

Suicidality Suicidality Suicidality Suicidality Suicidality

OCD OCD OCD OCD OCD

Concurrent

concerns

Concurrent

concerns

Concurrent

concerns

Concurrent

concerns

Concurrent

concerns

Age Age Depression Depression Depression

Personality

disorder Psychiatrist Psychiatrist

Legal

involvement

Legal

involvement

Legal

involvement

Youth

improvement

Page 84: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

71

6.4.3 M-M, M-O and O-O dyads: Satisfaction of modeling assumptions

In addition to examining relationships involving context variables, associations within and

between mechanisms and outcomes were also tested. The primary purposes of this step were to

satisfy modeling assumptions by a) ensuring mechanisms and outcomes were linearly related as

expected; and b) to identify the magnitude of risk of multicolinearity by comparing the relative

strength of M-M dyads to M-O dyads, since higher correlations within mechanisms than between

mechanisms and outcomes is an indicator of a high risk of multicolinearity, if all variables were

to be used in modelling (Tabachnick & Fidell, 2007). The results of this correlational analysis

are presented in Table 19.

Table 19. Spearman’s rho correlation coefficients for mechanism and outcome variables

Acc

essi

bil

ity

Conti

nuit

y

Fam

ily

involv

emen

t

FE

S F

FE

S S

m-B

CF

QoL

S

NA

VS

AT

Accessibility 1

Continuity 0.822 1

Family

involvement 0.774 0.828 1

FES F 0.303 0.301 0.407 1

FES S 0.217 0.223 0.325 0.642 1

m-BCFQoLS 0.411 0.444 0.463 0.539 0.367 1

NAVSAT 0.735 0.772 0.762 0.267 0.225 0.418 1

SRS 0.591 0.594 0.560 0.291 0.201 0.459 0.689

*All correlations have corresponding p-values < 0.001

6.4.3.1 M-M dyads

The results suggested that client scores on all three mechanisms – accessibility, continuity of

care, and family involvement - were significantly and strongly correlated with each other,

yielding three M-M dyads each coefficients ranging from 0.77 to 0.83 and corresponding critical

Page 85: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

72

alpha values of 0.000. Theoretically, high correlation between the mechanisms was anticipated

as these service characteristics complement each other in practice. Although these are three

distinct concepts with distinct measures in the literature, and every effort was made to ensure the

corresponding survey items had high face and content validity, it is possible that differences in

wording on the selected measures were perceived by clients as relatively minute, thus yielding

similar scores across the three measures. It is also possible that they were able to make the

distinction and were simply highly satisfied with all three mechanisms.

6.4.3.2 M-O dyads

The program theory presupposed that provision of accessible, continuous, family-inclusive

services would encourage family empowerment, improve FQOL, and ensure service satisfaction.

Since family empowerment and FQOL are theoretically more distal to the provision of certain

service characteristics than satisfaction with said service is, stronger correlations were expected

between mechanisms and satisfaction outcomes than with empowerment or FQOL outcomes.

Table 30 suggests that all three mechanisms were most strongly associated with NAVSAT score,

with coefficients somewhat consistent across the three, ranging from 0.74 to 0.78 and alpha

values of .000. The same was true for referred service satisfaction (0.56 < r < 0.59, p<0.001).

The associations between mechanisms and family empowerment and FQOL were more

moderate, with coefficients ranging from 0.22 to 0.46 (p<0.001).

6.4.4 Implications for modeling

Referring to back to Table 17, it was observed that there were no cases in which the correlations

between mechanisms and outcomes higher than the correlations within mechanisms. Higher

correlation between individual mechanisms than between mechanisms and outcomes is a strong

indication of likely multicolinearity, which violates a key assumption of inferential modeling

(Tabachnick & Fidell, 2007). Multicolinearity does not significantly impact overall model fit or

the actual parameter estimates, but is a concern because it decreases statistical power and at high

levels, can increase the variance of the model’s parameter estimates to the extent that they can

become unpredictable (so much so that signs can change) and problematic to interpret

(Studenmund, 2010).

Page 86: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

73

The results of the M-M and M-O correlational analyses indicated that, due to high risk of

multicolinearity, all three mechanism variables could not be individually reliably included in the

modeling of the outcome variables. To mitigate this issue, factor analysis was proposed as a data

reduction technique and is discussed in the next section.

6.5 Factor analysis

As identified above, the correlational analyses revealed that the associations between

mechanisms were always stronger than those between mechanisms and outcomes. This was

interpreted as evidence of a high risk of multicolinearity, which violates an assumption of

inferential modeling and can significantly undermine the interpretation of individual effects.

Despite the fact that the mechanisms were theorized to be three distinct concepts, the dataset did

not reflect this. Instead, it revealed similarly high scores, distributions, and correlation

coefficients amongst the three measures. Although they may have been genuinely independent

ratings, based on the current measures, they remained very likely to co-occur, so it was

considered a reasonable statistical solution to use a single variable representative of the

combined effects of navigation for modeling purposes (Yong & Pearce, 2013).

For this reason, a Principal Components Analysis was performed on the three mechanism

variables. Typically, large sample sizes greater than 300 are preferred for factor analyses,

however for datasets in which factor loadings are expected to be very high (>0.80), smaller

sample sizes are sufficient (Tabachnick & Fidell, 2007). In addition, only three items were

included, which more than satisfies Nunnally’s widely cited 10:1 cases per item rule (1978).

A Principal Components Analysis was performed on the three mechanism variables, the results

of which are presented below in Table 20. Principal Components was selected as the method of

extraction with a Direct Oblimin rotation for its primary use in data reduction and subsequent

modeling. A principal component (called “navigation mechanism”) with an eigenvalue of 2.714

was extracted; it explained 90.5% of the variance in scores across the three mechanisms. Since

only one component with an eigenvalue > 1 was extracted, rotation was not performed.

Communalities for all three variables were very high (0.885 < h2 < 0.927), indicating that all are

well represented by the extracted component. Component loadings for accessibility, continuity

and family involvement were 0.941, 0.963, and 0.950, respectively, suggesting each of the three

variables were very highly correlated with the extracted principal component. A scree plot

Page 87: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

74

confirmed the adequacy of a single component. The Kaiser-Meyer-Olkin statistic (0.763) is

relatively close to 1.0 and greater than the suggested 0.5 minimum, confirming sampling

adequacy; and the results of Bartlett’s Test of Sphericity were also significant (p=0.000),

confirming that the correlation matrix is not an identity matrix. Lastly, the determinant of the

correlation matrix was 0.052, which is significantly greater than 0.00001 and indicates that

multicolinearity is no longer a concern (Field, 2009; Tabachnik & Fidell, 2007; Yong & Pearce,

2013).

Table 20. Results of a Principal Components analysis for mechanism variables

Mechanism

variables

Communalities

(h2) C

om

pon

ents

Eig

en

valu

es

% o

f varia

nce

Com

pon

ent

load

ings

KM

O

Bartlett’s

Dete

rm

inan

t

Init

ial

Extr

acti

on

Accessibility 1.000 0.885 1 2.714 90.458 0.941

0.763 0.000 0.052

Continuity of

care 1.000 0.927 2 0.178 5.922 0.963

Family

involvement 1.000 0.902 3 0.109 3.621 0.950

The results of the Principal Components analysis suggested that a single component score in

place of the three mechanism variables was highly suitable for use in subsequent modeling. As

such, Bartlett’s factor (i.e. component) scores were calculated for each case and saved as new

variables in the dataset. Bartlett’s approach was most appropriate for the data since the manner in

which the scores were calculated (using maximum likelihood estimates) minimized the error

across the three mechanism variables and produced unbiased estimates that were most likely to

represent the true component scores (Yong & Pearce, 2013). Bartlett’s scores thus represented

the overall “navigation mechanism.” Descriptive statistics for the Bartlett’s scores are presented

in Table 21 below.

Page 88: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

75

Table 21. Descriptive statistics for the Bartlett Factor Score for “navigation mechanism”

Variable N Mean Median SD Min Max Skew Kurtosis K-S

Bartlett Factor

Score for

“navigation

mechanism”

134 0.00 0.36 1.0 -2.74 0.84 -1.32* 0.71 0.199**

*Statistic is more than twice its standard error **p<.01

To confirm the Bartlett’s scores were appropriately correlated with the outcome variables, a

quick correlational analysis was performed (Table 22). The results mimicked the patterns and

effect sizes observed when the mechanism variables were tested individually, confirming the

appropriateness of Bartlett’s scores for use in subsequent modeling. Therefore, the three

individual mechanisms variables were replaced with a representative component score –

“navigation mechanism” – in the inferential models that follow this section.

Table 22. Correlation coefficients for the Bartlett Factor Score with outcomes

Outcome

Family

empowerment FQOL Satisfaction

Variable FES F FES S m-

BCFQoLS NAVSAT SRS

Bartlett Mechanism Score

(“Navigation mechanism”) 0.368 0.296 0.458 0.812 0.623

* All correlations have corresponding p-values < 0.001

6.6 Inferential modelling

This study was guided by three broad research questions:

1. a) Who is the Family Navigation Project serving?

b) Is the Family Navigation Project reaching its target population?

c) Overall, are families satisfied with the services they received?

Page 89: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

76

2. a) Do families perceive the Family Navigation Project to be providing accessible,

continuous, family-inclusive care?

b) How does context influence perceived experience of the program?

3. a) Do families who perceive the Family Navigation Project as accessible, continuous and

family-inclusive experience better outcomes in terms of family empowerment, family

quality of life, and service satisfaction?

b) How does context influence these outcomes?

The first two research questions were addressed through descriptive and correlational analyses.

The third research question required an inferential approach that statistically tested the impact of

the navigation mechanism on each of the three outcomes of interest in the presence of the

associated contextual factors identified in the preceding correlational analyses.

6.6.1 Overview of statistical approach

Recall from Chapter 2 that RE is a theory-driven framework, and one of the goals of the

correlational analyses was to identify dyads and triads of variables that would help explain the

influence of context and mechanisms on outcomes. The next step in the framework was to test

these C-M-O configurations for statistical significance in a series of generalized linear models

intended to support the proposed program theory. Since only one model per dependent variable

was tested, adjustments for multiple analyses were not required and significance was confidently

interpreted at a critical alpha level of 0.05.

The study was designed to yield a set of continuous dependent variables for use in generalized

linear modelling. This was the preferred approach as generalized linear models allow for

specification of the probability distribution of the dependent variable, and at least some of the

dependent variables in this study were anticipated to be significantly skewed (McCullagh &

Nelder, 1989). The results of the descriptive analysis did indicate that while the FES Family and

FES Service-seeking subscales and m-BCFQoLS scores could be fit with a normal distribution,

the two satisfaction outcomes were very highly negatively skewed. As such, generalized linear

models were selected and run for all dependent variables as they can fit different distributions

while producing consistent statistical output across outcomes. Different distributions are fit using

a canonical link function, which relates the mean of the response variable to the linear predictors

Page 90: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

77

in the model. Identity links are the most basic link function and can be used in any case where

transformation is not required, which is most commonly in the case of normal and gamma

distributions (McCullagh & Nelder, 1989). Other link functions are transformative; examples

include log links, for use when the response variable cannot be negative (e.g. Poisson counts),

and logit and probit links, for use when data are binary.

For the three normally distributed outcome variables (i.e. FES Family, FES Service-seeking and

m-BCFQoLS scores), generalized linear models can be used by specifying a normal distribution

and an identity link. This procedure produces identical coefficients to those produced by

traditional linear regression; the main difference is in the specific significance tests used (F-test

for linear regression, Wald Chi-Square for generalized linear models) (McCullagh & Nelder,

1989).

Descriptive analyses of the normality statistics, Q-Q and P-P plots, as well as literature on the

topic, suggested that the satisfaction outcomes were highly negatively or left-skewed and could

not be reliably fit with a normal distribution. For generalized linear modelling of non-normal

outcomes (i.e. satisfaction scores), more appropriate distributions can be specified. Literature

suggests satisfaction data can be fit with a beta distribution, which can take on many shapes

provided appropriate shape parameters are applied; however, shape parameters can be difficult to

estimate (Lindsey & Jones, 1998; McCullagh & Nelder, 1989). The second option was to apply a

gamma distribution, which is the inverse of a beta distribution; visually, it looks like a highly

positive or right-skewed distribution (Hardin & Hilbe, 2007; McCullagh & Nelder, 1989). As

such, to apply a gamma distribution to negative or left-skewed satisfaction scores, scores would

need to be inverted. Gamma distributions typically employ either an identity or log link. For the

current study, a gamma distribution with an identity link was determined to be most appropriate

by the PI, a choice that was confirmed as appropriate following consultation with a statistician at

the University of Toronto.

This latter approach (i.e. applying a gamma distribution) was considered preferable over risking

inappropriately specified shape parameters for a beta distribution; and also preferable over

categorizing the variables and using a logistic regression or gamma distribution with a log link

for several reasons. First, categorizing negates the design efforts to yield a continuous dependent

variable, which inherently has more information. Second, the heavily skewed distribution means

Page 91: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

78

the naturally emerging categories would be highly unbalanced unless condensed into even larger

categories, losing even more information. Third, categories with smaller sample sizes lack the

statistical power needed to include the proposed predictors. Lastly, an identity link was selected

over a model using the log link because the dependent variable was positive and continuous and

did not need to be further log-transformed, which would unnecessarily complicate interpretation

of the regression coefficients for categorical variables with more than one level (Lindsey &

Jones, 1998). Therefore, following a statistical consultation, inverse scores were calculated by

subtracting the original score from the denominator plus one, and then saved as new variables.

These new variables then served as the dependent variable for satisfaction, or dissatisfaction; that

is, use of the inverse score means the applied generalized linear model models dissatisfaction.

For these models, a gamma distribution was specified along with an identity link.

There are several important differences to note in the accompanying statistics produced by

generalized linear models (Hardin & Hilbe, 2007; McCullagh & Nelder, 1989). First,

standardized beta coefficients were not available as they would have been through the linear

regression procedure so effect sizes of individual predictors are less easily compared. Second,

whereas the linear regression procedure calculates variance explained (R2), most of the goodness

of fit statistics produced by the generalized linear model procedure are less easily interpreted in

this particular study. For example, the procedure produces deviance and Pearson chi-square

statistics, which are known to be sensitive to responses equal to zero (that occur often in dummy

coded categorical covariates, such as the ones included in this study) and empty cells (that often

occur with continuous covariates, such as the ones included in this study). Instead, the selected

approach was to report the results of the Omnibus Test of model fit for each dependent variable.

This procedure tested the complete fitted model against the null model using a likelihood ratio

chi-square and corresponding critical alpha value for significance testing. Models for each

dependent variable were then compared by the magnitude of the likelihood ratio chi-square

value, where higher values indicate better model fit.

6.6.2 Testable hypotheses

At the outset of this study, several broad hypotheses were proposed in relation to each of the

outcomes of interest (Table 2, Section 3.1). Following measurement specification and the

confirmation of covariates for each outcome in Section 6.4.2, hypotheses could then be further

Page 92: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

79

specified prior to modelling. The specific hypotheses for each outcome variable are presented in

Table 23 below.

Table 23. Dependent variables and corresponding hypotheses

Dependent variable Specific hypotheses

FES Family

H1: Clients’ “navigation mechanism” score positively predicts clients’

FES Family scores even when selected contextual covariates are

included

FES Service-seeking

H1: Clients’ “navigation mechanism” score positively predicts clients’

FES Service-seeking scores even when selected contextual covariates

are included

m-BCFQoLS

total score

H1: Clients’ “navigation mechanism” score positively predicts clients’

m-BCFQoLS total scores even when selected contextual covariates are

included

Inverse NAVSAT

total score

H1: Clients’ “navigation mechanism” score negatively predicts

clients’ NAVSAT total scores even when selected contextual covariates

are included

Inverse Satisfaction

with referred

services (SRS) score

H1: Clients’ “navigation mechanism” score negatively predicts

clients’ satisfaction with referred services (SRS) scores even when

selected contextual covariates are included

As a reminder, Table 24 presents a summary of the covariates selected for inclusion, which were

grounded in the program theory and confirmed for inclusion by the results of the correlational

analysis (refer back to Section 6.4.2). “Navigation mechanism” factor score was the primary

predictor for all outcomes. Contextual covariates were included because they were expected to

influence the mechanism and/or outcomes. However, the navigation mechanism was

hypothesized to maintain an independent effect on the outcomes of interest even when selected

contextual covariates were included.

Page 93: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

80

Table 24. Covariates selected for inclusion in modeling by outcome

Dependent

variable

Family empowerment Family

quality of life Satisfaction

FES F FES S m-

BCFQoLS

Inverse

NAVSAT

total score

Inverse

SRS score

Mechanism

(i.e. predictor)

“Navigation

mechanism”

factor score

“Navigation

mechanism”

factor score

“Navigation

mechanism”

factor score

“Navigation

mechanism”

factor score

“Navigation

mechanism”

factor score

Contextual

covariates

No. contacts with

Navigator

No. contacts

with Navigator

No. contacts

with Navigator

No. contacts

with Navigator

No. contacts

with Navigator

Suicidality Suicidality Suicidality Suicidality Suicidality

OCD OCD OCD OCD OCD

Concurrent

concerns

Concurrent

concerns

Concurrent

concerns

Concurrent

concerns

Concurrent

concerns

Age Age Depression Depression Depression

Personality

disorder

Psychiatrist

referral

Psychiatrist

referral

Legal

involvement

Legal

involvement

Legal

involvement

Youth

improvement

Results are discussed by outcome below.

6.6.3 Models of family empowerment

Recall that the third initial guiding research questions in this study asked, “What is the impact of

the Family Navigation Project on family empowerment and quality of life? What components are

most important, for whom, in which circumstances, and why?” Descriptive and correlational

analyses led to the identification of predictors of empowerment and covariates of mechanisms,

which were used to form specific hypotheses about the context and mechanism variables that

best predict FES Family and FES Service-seeking scores. The two hypotheses related to these

two dependent variables were tested in generalized linear models. Results are reported below.

Page 94: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

81

6.6.3.1 FES Family subscale

The specific hypothesis related to family empowerment at the family level was:

H1: Clients’ “navigation mechanism” scores positively predict FES Family scores even when

contextual covariates are included.

Navigation services are designed to help clients overcome real and perceived barriers to care,

which in this model, are represented by the contextual factors found to be most strongly

associated with the mechanism and outcome (FES Family): number of contacts with a navigator,

OCD, suicidality, concurrent concerns, and age. If navigation was provided as intended, the

coefficient for “navigation mechanism” score should be statistically significant even when the

associated context variables are included in the model. Results are presented in Table 25 below.

Table 25. Parameter estimates for the dependent variable “FES Family score”

DV – FES Family score (n=134)

Parameter B SE CI < 95%< CI Wald

Chi-square df p-value

Intercept 44.19 2.87 38.56, 49.82 236.38 1 .000

Mechanism factor score 2.06 0.56 0.96, 3.16 13.40 1 .000*

No. contacts with Navigator -0.02 0.04 -0.09, 0.05 0.39 1 .535

Suicidality 3.04 1.52 -0.05, 6.02 3.98 1 .046*

OCD -0.02 1.57 -3.10, 3.07 0.00 1 .992

Concurrent concerns -2.26 1.10 -4.41, -0.10 4.21 1 .040*

Age 12-15 5.53 2.59 0.46, 10.61 4.57 1 .033*(.178)a

16-19 1.57 1.73 -1.83, 4.98 0.82 1 .365(178)a

20-24 1.01 1.67 -2.27, 4.28 0.36 1 .546(.178)a

25+ (ref)

*p<.05 aIndicates significance level of variable main effect

A generalized linear model specifying a normal distribution and an identity link was performed

where FES Family subscale score was the dependent variable, navigation mechanism score was

Page 95: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

82

the primary predictor, and number of contacts with a Navigator, suicidality, OCD, concurrent

concerns and age were all included as covariates. The results suggested that navigation

mechanism score significantly, positively predicted FES Family score (B=2.06, SE=0.56, 0.96 <

95% CI < 3.16; Wald 2=13.40, p=0.000) even when the aforementioned context variables were

included. That is, for every unit increase in navigation mechanism score, FES Family scores

increased by 1.97 units. A relatively small standard error and narrow confidence intervals

suggested this coefficient estimate was statistically reliable.

Several context variables were also significantly predictive of the dependent variable. Following

from the results of the correlational analysis, the results of the inferential model suggested that

suicidality significantly, positively predicted individual FES Family scores in this sample

(B=3.04, SE=1.52, -0.05 < 95% CI < 6.02, Wald 2=3.98, p=0.046). In contrast, reported

concurrent MHA concerns were found to significantly, negatively predict FES Family scores in

this sample (B=-2.26, SE=1.10, -4.41 < 95% CI < 0.10, Wald 2=4.21, p=0.040). The main

effect of age was not significantly predictive, meaning as a whole, age does not significantly

influence family empowerment scores (Wald 2=4.92, p=0.178). The results did indicate that

relative to the reference group, which was youth aged 25+, clients with youth aged 12 to 15 were

significantly more likely to score higher on family-level family empowerment. However, only

nine youth in this sample fell into the aged 12 to 15 category.

For each variable, the Wald Chi-square statistic indicated the magnitude and significance of the

main effect size (scaled using the estimated standard error). The results suggested that the

navigation mechanism score had over three times the effect relative to the contextual predictors;

and that concurrent concerns and suicidality still modest effects - negative and positive,

respectively. In comparison, the effect sizes for OCD and number of contacts with a Navigator

were minimal in both magnitude and significance.

Overall model fit was assessed using the results of the Omnibus Test of Model Fit, which uses a

likelihood ratio chi-square test (G test) to compare the fit of two models, in this case the null

model (with no predictors) and the model fitted with the theorized predictors above. The

likelihood ratio chi square test compares the log likelihoods of both models; higher log

likelihoods (distributed chi square) indicate better model fit, and better model fit indicates a

greater proportion of explained variance in response data. If the test is statistically significant

Page 96: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

83

(i.e. p<.05), the model fitted with predictors is confirmed to fit the data significantly better than

the null model (Hardin & Hilbe, 2007).

The Omnibus Test of Model Fit for the current model indicates that the fitted model performed

significantly better than the null model (LR 2=34.03, df=8, p=0.000); that is, when the proposed

predictors were included, the model was better able to predict individual FES Family scores than

when predictors were not included.

6.6.3.2 FES Service-seeking subscale

The specific hypothesis related to family empowerment at the level of youth service seeking was:

H1: Clients’ “navigation mechanism” scores positively predict FES Service-seeking scores even

when contextual covariates are included.

Program theory and correlational analysis confirmed a number of theorized contextual predictors

to be significantly related to the outcome, in this case, FES Service-seeking scores. As such, the

following contextual variables were included as covariates in the model where FES Service-

seeking score was the dependent variable: number of contacts with a Navigator, OCD,

suicidality, age, and concurrent MHA concerns. If navigation was provided as intended, the

coefficient for “navigation mechanism” score should be statistically significant even when the

associated context variables mentioned are included in the model. Results are presented in Table

26 and interpreted below.

Table 26. Parameter estimates for the dependent variable “FES Service-seeking score”

DV – FES Service-seeking score (n=134)

Parameter B SE CI < 95%< CI Wald

Chi-square df p-value

Intercept 48.90 2.86 43.3, 54.51 292.91 1 .000

Mechanism factor score 1.52 0.74 0.68, 2.97 4.21 1 .040*

No. contacts with Navigator 0.01 0.05 -0.83, 0.11 0.08 1 .777

Suicidality 5.45 2.01 1.15, 9.38 7.38 1 .007*

OCD 2.97 2.07 -1.10, 7.04 2.05 1 .152

Page 97: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

84

Concurrent concerns -3.19 1.44 -6.03, -0.35 4.86 1 .027*

Age 12-15 7.61 3.41 0.92, 14.29 4.97 1

.026*

(.158)a

16-19 2.88 2.29 -1.61, 7.36 1.58 1 .209

(.158)a

20-24 1.97 2.20 -2.35, 6.28 0.80 1 .371

(.158)a

25+ (ref)

*p<.05 a Indicates significance level of variable main effect

A generalized linear model specifying a normal distribution and an identity link was performed

where FES Service-seeking subscale score was the dependent variable, navigation mechanism

score was the primary predictor, and number of contacts with a Navigator, suicidality, OCD,

concurrent concerns and age were all included as covariates. The results suggested that

navigation mechanism score significantly, positively predicted FES Service-seeking score

(B=1.52, SE=0.74, 0.68 < 95% CI < 2.97; Wald 2=4.21, p=.040) even when the aforementioned

context variables were included. That is, for every unit increase in navigation mechanism score,

FES Service-seeking score increased by 1.52 units.

In this model, several context variables were also found to be significantly predictive of FES

Service-seeking scores. As with FES Family scores, concurrent MHA concerns predicted poorer

outcomes (B=-3.19, SE=1.44, -6.03 < 95% CI < -0.35; Wald 2=4.86, p=.027), while suicidality

predicted better outcomes (B=5.45, SE=2.01, 1.15 < 95% CI < 9.38; Wald 2=7.38, p=.007).

OCD and number of contacts with a Navigator were non-significant in this model as well. The

main effect of age was again not significantly predictive (Wald 2=5.19, p=0.158); however,

again relative to the reference group, youth aged 25 and older, clients with youth aged 12 to 15

were significantly more likely to score higher. Again however, only nine clients had youth aged

12 to 15.

Comparing the magnitude of the Wald Chi-square statistics suggested that the navigation

mechanism score in this model (2=4.21) had less of an effect overall and relative to context than

in the FES Family model (2=13.40). The standard error estimates were also slightly larger and

the confidence intervals were slightly wider, suggesting the estimates within the FES Service-

Page 98: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

85

seeking model are slightly less reliable than those in the FES Family model. However, the

Omnibus Test of model fit indicates that the fitted model still performs significantly better than

the null model (LR 2=32.34, df=8, p=0.000); that is when the proposed predictors were

included, the model was better able to predict individual FES Service-seeking scores than when

predictors were not included. The magnitude of the likelihood ratio (distributed chi square)

indicates that the model of FES Service-seeking scores was similar in overall fit to the FES

Service-seeking model.

6.6.4 Model of family quality of life

The specific hypothesis related to FQOL was as follows:

H1: Clients’ “navigation mechanism” scores positively predict m-BCFQoLS total scores even

when contextual covariates are included.

The program theory proposed the following contextual factors to influence FQOL, which were

confirmed to be related to the outcome variable (m-BCFQoLS total score) in the correlational

analysis and included in the current model as covariates: number of contacts with a Navigator,

OCD, suicidality, depression, personality disorders, concurrent MHA concerns and legal

involvement. If navigation was provided as intended, the coefficient for “navigation mechanism”

score should be statistically significant even when the covariates mentioned are included in the

model. Results are presented in Table 27 and interpreted below.

Table 27. Parameter estimates for the dependent variable “m-BCFQoLS total score”

DV – m-BCFQoLS total score (n=131)

Parameter B SE CI < 95%< CI Wald

Chi-square df p-value

Intercept 73.83 2.34 69.25, 78.14 997.22 1 .000

Mechanism factor score 4.95 1.20 2.80, 7.12 20.32 1 .000*

No. contacts with Navigator 0.04 0.07 -0.10, 0.17 0.29 1 .588

Suicidality 6.59 2.70 1.30, 11.88 5.97 1 .015*

OCD -4.06 2.84 -9.62, 1.50 2.05 1 .152

Page 99: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

86

Concurrent concerns -4.73 2.15 -8.94, -0.53 4.86 1 .027*

Depression 3.14 2.02 -0.82, 7.09 2.41 1 .121

Personality disorder -7.55 3.53 -14.46, -0.64 4.59 1 .032*

Legal involvement -4.96 2.48 -9.81, -0.10 4.01 1 .045*

*p<.05

A generalized linear model specifying a normal distribution and an identity link was performed

where m-BCFQoLS total score was the dependent variable, navigation mechanism score was the

primary predictor, and number of contacts with a Navigator, suicidality, OCD, depression,

personality disorders, concurrent MHA concerns and legal involvement were all included

covariates. The results suggested that navigation mechanism score significantly, positively

predicted m-BCFQoLS total score (B=4.95, SE=1.20, 2.80 < 95% CI < 7.12; Wald 2=20.32,

p=.000) even when the aforementioned context variables were included. That is, for every unit

increase in navigation mechanism score, FQOL total score increased by 4.95 units.

In this model, four context variables were also significantly predictive of m-BCFQoLS total

scores. As with the FES models, concurrent MHA concerns again predicted poorer outcomes

(B=-4.73, SE=2.15, -9.62 < 95% CI < 1.50; Wald 2=4.86, p=.027), while suicidality predicted

better outcomes (B=6.59, SE=2.70, 1.30 < 95% CI < 11.88; Wald 2=5.97, p=.015). In this

model, personality disorder concerns were found to be significantly predictive of poorer FQOL

total scores (B=-7.55, SE=3.53, -14.46 < 95% CI < -0.64; Wald 2=4.59, p=.032), as was legal

involvement in cases (B=-4.96, SE=2.48, -9.81 < 95% CI < -0.10; Wald 2=4.01, p=.045).

Comparing the magnitude of the Wald Chi-square statistics suggested that the navigation

mechanism score in this model had stronger effect overall and relative to the context variables

than both FES models. The Omnibus Test of model fit also indicated that the fitted model still

performed significantly better than the null model (LR 2=63.14, df=8, p=0.000); that is when

the proposed predictors were included, the model was better able to predict m-BCFQoLS total

scores than when predictors were not included. The magnitude of the likelihood ratio (distributed

chi square) indicates that the model of FQOL performed better in overall fit compared to either

of the FES models.

Page 100: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

87

6.6.5 Models of service satisfaction

Service satisfaction was evaluated using the FNP’s in-house navigation satisfaction tool, which

captures both satisfaction with navigation services and the services to which families were

referred. It was theorized that satisfaction with navigation services would be the most proximal

outcome of interest in this study. Modelling results would lend support to this theory if the

influence of the navigation mechanism score is significantly greater in effect size than any of the

other modelled outcomes. As discussed previously, inverse satisfaction scores were modelled

using generalized linear models with specified gamma distributions and identity links. The

dependent variable was therefore dissatisfaction with navigation services and dissatisfaction with

referred services.

6.6.5.1 Inverse NAVSAT total score

The specific hypothesis related to satisfaction with navigation services was:

H1: Clients’ “navigation mechanism” scores negatively predict inverse NAVSAT total scores

even when contextual covariates are included.

The following contextual factors were theorized to influence satisfaction and confirmed to be

related to the outcome variable in the correlational analysis: number of contacts with a

Navigator, OCD, suicidality, concurrent concerns, depression, legal involvement and reason for

contact being need for a psychiatrist. These contextual factors were included as covariates in the

current model. If navigation was provided as intended, the coefficient for “navigation

mechanism” score should be statistically significant even when the covariates mentioned are

included in the model. Results are presented in Table 28 and interpreted below whereby each

coefficient represents the influence on increasing or decreasing dissatisfaction with navigation

services.

Table 28. Parameter estimates for the dependent variable “Inverse NAVSAT total score”

DV – Inverse NAVSAT total score (n=130)

Page 101: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

88

Parameter B SE CI < 95%< CI Wald

Chi-square df p-value

Intercept 3.99 0.32 3.36, 4.61 154.99 1 .000

Mechanism factor score -3.12 0.31 -3.70, -2.51 103.18 1 .000*

No. contacts with Navigator 0.00 0.00 -0.01, 0.00 0.88 1 .348

Suicidality -0.15 0.18 -0.51, 0.21 0.67 1 .414

OCD 0.49 0.58 -0.64, 1.62 0.73 1 .394

Legal involvement 0.61 0.30 0.02, 1.19 4.12 1 .042*

Concurrent concerns 0.14 0.18 -0.21, 0.49 0.65 1 .421

Psychiatrist referral -0.28 0.18 -0.62, 0.07 2.48 1 .115

Depression -0.11 0.20 -0.51, 0.28 0.314 1 .575

*p<.05

A generalized linear model specifying a gamma distribution and an identity link was performed

where inverse NAVSAT total score was the dependent variable, navigation mechanism score

was the primary predictor, and the number of contacts with a Navigator, OCD, suicidality,

concurrent concerns, legal involvement, and reason for contact being need for a psychiatrist

referral were all included as covariates. The results indicated that navigation mechanism score

significantly, strongly, negatively predicted dissatisfaction with navigation (B=-3.12, SE=0.31, -

3.70 < 95% CI < -2.51; Wald 2=103.18, p=.000) even when the aforementioned context

variables were included. That is, for every unit increase in navigation mechanism score, inverse

NAVSAT total score decreased by 3.12 units. In other words, as navigation mechanism score

increased, dissatisfaction with navigation decreased; thus, navigation mechanism positively

predicted satisfaction with navigation.

In this model, only one context variable was significantly predictive of the dependent variable.

Legal involvement in a client’s case was positively predictive of dissatisfaction (B=0.61,

SE=0.30, 0.02 < 95% CI < 1.19; Wald 2=4.12, p=.042). That is, clients with legal involvement

in their cases were significantly less likely to be satisfied with navigation.

Page 102: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

89

Comparing the magnitude of the Wald Chi-square statistics suggests that the navigation

mechanism score in this model had the strongest effect overall and relative to the context

variables observed to date; the statistic was nearly 300% greater than the next highest statistic

(FQOL model) and had much less distance between it and the intercept statistic, suggesting

better model. The standard error estimates were also significantly smaller relative to their

estimates, and the confidence intervals were much narrower, suggesting the estimates within the

inverse NAVSAT total score model were highly statistically reliable. Finally, the Omnibus Test

of model fit confirms that the fitted model still performed significantly better than the null model

(LR 2=170.04, df=8, p=0.000); that is when the proposed predictors were included, the model

was better able to predict individual inverse NAVSAT total scores than when predictors were not

included. The magnitude of the likelihood ratio (distributed chi square) indicates that the

NAVSAT model performed exponentially better in overall fit compared to FES and FQOL

models.

6.6.5.2 Inverse satisfaction with referred services (SRS) score

The specific hypothesis related to satisfaction with referred services was:

H1: Clients’ “navigation mechanism” scores negatively predict inverse SRS scores even when

contextual covariates are included.

In this model, the following contextual factors were included as covariates: number of contacts

with a Navigator, OCD, suicidality, concurrent concerns, depression, legal involvement, youth

improvement, and reason for contact being need for a psychiatrist referral. If navigation was

provided as intended, the coefficient for “navigation mechanism” score should be statistically

significant even when the covariates mentioned are included in the model. Results are presented

in Table 29 and interpreted below whereby each coefficient represents influence on increasing or

decreasing dissatisfaction with referred services.

Table 29. Parameter estimates for the dependent variable “Inverse satisfaction with

referred service (SRS) score”

DV – Inverse satisfaction with referred services (SRS) (n=129)

Page 103: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

90

Parameter B SE CI < 95%< CI Wald

Chi-square df p-value

Intercept 3.59 0.51 2.60, 4.59 50.33 1 .000

Mechanism factor score -0.85 0.18 -1.20, -0.49 21.55 1 .000*

No. contacts with Navigator -0.01 0.01 -0.02, 0.00 1.67 1 .196

Suicidality -0.36 0.21 -0.93, -0.03 4.37 1 .089

OCD 0.80 0.42 -0.03, 1.62 3.53 1 .060

Concurrent concerns 0.17 0.21 -0.24, 0.57 0.65 1 .419

Depression -0.19 0.21 -0.60, 0.22 0.82 1 .366

Psychiatrist referral -0.39 0.19 -0.77, 0.02 4.17 1 .041*

Legal involvement 0.12 0.26 -0.40, 0.63 0.20 1 .656

Youth

improvement

Improved 0.42 0.55 -0.66, 1.50 0.58 1

.446

(.001)*a

Stayed the

same -0.66 0.49 -1.61, 0.29 1.85 1

.174

(.001)*a

Worsened

(ref)

*p<.05 aIndicates significance level of variable main effect

A generalized linear model specifying a gamma distribution and an identity link was performed

where inverse SRS score was the dependent variable, navigation mechanism score was the

primary predictor, and the number of contacts with a Navigator, OCD, suicidality, concurrent

concerns, depression, legal involvement, youth improvement, and reason for contact being need

for a psychiatrist referral were all included covariates. These variables were included as

covariates because they were theorized to influence satisfaction with referred services, and were

confirmed to be significantly related to the outcome variable in the correlational analysis. The

results suggested that navigation mechanism score significantly, modestly, negatively predicted

dissatisfaction with referred services (B=-0.85, SE=0.18, -1.20 < 95% CI < -0.49; Wald

2=21.55 p=.000) even when the aforementioned context variables were included. That is, for

every unit increase in navigation mechanism score, inverse SRS score decreased by 0.85 units. In

Page 104: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

91

other words, as navigation mechanism score increased, dissatisfaction with referred services

decreased; thus, navigation mechanism positively predicted satisfaction with referred services.

In this model, only two context variables were significantly predictive of the dependent variable.

This time, the need for a psychiatrist referral negatively predicted dissatisfaction with referred

services (B=-0.39, SE=0.19, -0.77 < 95% CI < 0.02; Wald 2=4.17, p=.041). That is, clients who

contacted the FNP in need of a psychiatrist were significantly more likely to be satisfied with

referred services. The model also indicated that although perceived youth improvement had a

significant main effect on the outcome (Wald 2=13.41, df=2, p=.001). Referring to Table 29,

the positive and negative signs for each category suggested that families with no change in youth

perceived functional status were less likely to be dissatisfied than those whose youth had

worsened, while families who perceived youth improvement were counter-intuitively more likely

to be dissatisfied. These differences aligned with the counter-intuitive inverse relationship

between improvement and satisfaction revealed in the correlational analysis. However, in the

current inferential analysis, differences between categories were not found to be statistically

significant.

Comparing the magnitude of the Wald Chi-square statistics suggested that the navigation

mechanism score in this model was much closer in effect size to the FES and FQOL models than

it was to the inverse NAVSAT total score model. The standard error estimates were still small,

and the confidence intervals were still narrower, suggesting the estimates were reliable.

The Omnibus Test of model fit confirmed that the fitted model performed significantly better

than the null model (LR 2=80.84, df=10, p=0.000); that is when the proposed predictors were

included, the model was better able to predict individual inverse satisfaction with referred

services (SRS) scores than when predictors were not included. The magnitude of the likelihood

ratio (distributed chi square) indicates that the model of SRS scores fit better than the FES

models, and was similar in fit to the FQOL model. However, overall model fit for SRS was not

nearly as robust as it was for the inverse NAVSAT total score model. This aligns with the

premise that NAVSAT score was the most proximal outcome measure of those selected in this

study.

Page 105: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

92

Table 30 presents the overall model fit statistics by outcome below and Table 31 presents the

final model equations by dependent variable. Overall, the results support the conclusion that

family navigation, as defined by the navigation mechanism score, has a significant, positive

impact on each of the outcomes of interest in this study.

Table 30. Omnibus Test of Model Fit results by dependent variable

Omnibus Test Likelihood

ratio chi-

square

df p-value

DV

FES F 34.03 8 0.000

FES S 32.34 8 0.000

m-BCFQoLS 63.14 8 0.000

NAVSAT 170.04 8 0.000

SRS 80.84 10 0.000

Table 31. Generalized linear model equations by dependent variable

Dependent

variable Equation

FES

Family

= 44.19 + 2.06(navigation mechanism) + 3.04(suicidality) -2.26(concurrent

concerns)

FES

Service-

seeking

= 48.90 + 1.52(navigation mechanism) + 5.45(suicidality) -3.19(concurrent

concerns)

m-

BCFQoLS

= 73.8 + 4.95(navigation mechanism) + 6.59(suicidality) -4.73(concurrent

concerns) -7.55(personality disorder) – 4.96(legal involvement)

Inverse

NAVSAT

total score

= 3.99 -3.12(navigation mechanism) + 0.61(legal involvement)

Inverse

SRS score = 3.59 -0.85(navigation mechanism) - 0.39(psychiatrist referral)

Page 106: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

93

Chapter 7 Qualitative Analysis

Overview of qualitative approach

During administration of the online survey, participants were prompted at the end of each section

to add any additional qualifying information, points of interest, and/or to further expand upon

their experiences with the FNP in unlimited open text fields. As with responses to the

quantitative measures, providing additional qualitative data was entirely voluntary. Qualitative

data was collected and merged with quantitative data to generate rich, qualitative thematic

descriptions that would expand upon and enhance the program theory and quantitative findings,

particularly with regard to the influence of context. Descriptive qualitative analyses are an

effective and often undervalued means of supplementing and better understanding a dataset. It is

an approach that borrows many techniques from many other approaches, and falls somewhere

between the limits of quantitative inference and more in-depth and rigorous qualitative

approaches, such as grounded theory or phenomenology (Sandelowski, 2000). The goal of

descriptively analyzing the open-ended qualitative data was simply to further contextualize and

improve understanding of the data.

Of the 134 survey participants, 71 (or 53.0% of the study sample) provided complementary

qualitative data that corresponded to at least one context, mechanism and/or outcome of interest.

Qualitative data from these 71 unique sources were imported, coded and analyzed using QSR

International’s NVivo 11.4 for Mac, a software package supporting qualitative and mixed

methods research (2015). Once the qualitative data was imported, contextual quantitative data

from the survey and chart review phases were also imported to create a merged dataset for

thematic analysis. Logistically, this was accomplished in NVivo using case classifications and

attributes, a feature which allows the user to record categories of descriptive information (like

demographics or type of MHA concern) about each client. Users can then view themes and

coded phrases within and across classifications and themes, producing a variety of C-M-O

configurations.

These qualitative C-M-O configurations further contributed toward understanding the

experiences and outcomes of families in this study, especially in relation to contextual influences

Page 107: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

94

that were less easily captured and/or elucidated with quantitative measures. For example,

systemic-level contexts such as long wait lists or consent and capacity legislation, were factors

theorized to influence experience and outcomes, but which were difficult to pre-define and

measure quantitatively. With regard to individual-level contexts, a range of factors, such as youth

age and whether youth had concurrent mental health and addiction concern, were theorized to

influence experience and outcomes. Of the theorized contextual influences, only approximately

30.0% were found to be statistically significantly associated with experience and outcomes in the

correlational analysis. Of those that contextual influences that were statistically significantly

associated, only approximately 33.0% were found to be statistically significantly predictive in

inferential modeling of outcomes. However, the program theory maintained that such contextual

influences were very likely to have played a role; it is possible that the degree of influence was

not to the extent that statistical significance could be detected in the current sample. Qualitative

C-M-O configurations were thus helpful in identifying if and describing how and to what extent

context may have influenced a family’s experience and outcomes.

Since the RE framework is theory-driven, it presupposes which key contexts, mechanisms and

outcomes of interest are likely to be present in the qualitative data (i.e. those identified in the

program theory and background research). However, RE is also meant to adapt to emergent

themes, and so any additional contexts, mechanisms and outcomes outside the program theory

were also recorded (Pawson & Tilley, 1997). NVivo is an ideal analytical software for this

purpose as it allows for users to code phrases of data according to pre-identified themes (or

“nodes”), as well as allow for groupings of similar phrases to accumulate under emergent

themes.

To begin, phrases were first coded by domain as related to either contexts (C), mechanisms (M),

or outcomes (O). Phrases describing a condition existing external to or prior to enrolment with

the FNP were coded as contextual (C); mechanisms were coded when phrases spoke directly to

program activities and resources (M); and outcomes were coded when phrases spoke to a direct

result of a client’s experience with the FNP (Pawson & Tilley, 1997). With the integrated

dataset, each code also had a case classification and a corresponding set of attributes from the

quantitative data (i.e. age, gender, type of concerns). As mentioned above, the purpose of the

qualitative data was to generate evidence for additional C-M-O configurations that would add to

a more robust understand of a family’s experience and outcomes, particularly in terms of further

Page 108: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

95

describing contextual influences (i.e. if, to what extent, and how context exerted influence on

experience and/or outcomes). Codes were then grouped under proposed and emergent themes

related to how context influenced experiences and outcomes, and hierarchies were assembled

where theoretically appropriate (Byng, et al., 2005). To ensure confidentiality, clients were

identified by number only and all names have been changed. Themes are discussed in depth by

domain – context, mechanisms, or outcomes - below.

7.1 Descriptive and thematic analyses

7.1.1 Context

Based on the program theory and background research, several contextual themes known to

impact this population at different levels were proposed. At the individual level, most theorized

contextual factors such as demographics and illness characteristics were collected from closed-

ended survey questions and client charts. However, there are some individual-level contexts that

were theorized to influence experience and outcomes, but which were not found to be

statistically significant in effect, or which were not easily captured quantitatively. The qualitative

data spoke to one theme in particular that relates to individuality, the nature of MHA conditions,

and the fact that the client in this program is the family member, not the youth.

7.1.1.1 Theme 1: You can’t force someone to get well

Of the 71 clients who provided qualitative data, over a quarter (n=19, or 26.8%) spoke to the fact

that their experience and/or potential for positive outcomes was hindered by the youth in

question’s unwillingness or inability to get well. The single most commonly associated

contextual variable under this theme was age. Most often, this theme related to the youth being

of adult age and in control of their own treatment, and/or generally unwilling to engage in the

help-seeking and treatment processes. Many youth in this sample were over the age of 19 and are

technically adults responsible their own behaviour; 16 of 19 clients who were coded on this

theme had youth who were aged 19 and older at the time of seeking help. Clients explained how

this affected their experience: “My daughter is 25 years old so I have little control about what

services she will accept or decline. I feel that at this point I can only offer her suggestions and

options for her to help herself” (10059); and, “…it is difficult because our son is 19. It is up to

Page 109: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

96

him how he decided to utilize the services that are presented to him. The drugs have got [sic]

worse and he is no longer living with us or accessing any services” (10016); and,

“The concept is excellent and initially was helpful. Because my nephew was 21 years old

and living in youth hostels, it was necessary for him to be engaged in getting help.

Despite the efforts of the navigator and myself, it’s not clear that [Tim] was very engaged

... as [Tim] showed limited commitment to seeking help, the role of the navigator

lessened” (10011).

Younger age did not guarantee engagement though, as even younger youth have the right to

refuse treatment in Canada. One client noted, “Unfortunately all the service recommendations

could not work because our teen refuses meds and refuses counselling or psychological support”

(10068); another, whose child is only 16, “My child refuses to seek treatment and doesn’t think

there is anything wrong with him” (10082); and another, “Since there is no way for parents to

force their children to go or stay in treatment in Canada, he never wanted to, or did” (10123).

Type of MHA concern mattered for families. Different MHA conditions can manifest uniquely

and vary significantly in nature, particularly in terms of disability, chronicity and responsiveness

to treatment (Kessler, et al., 2005, 2012). For example, depression is highly treatable and

associated with positive outcomes, whereas personality disorders and OCD are chronic severe

conditions that are difficult to treat and for which fewer resources are available. Clients who

spoke to this idea of being unable to force their children to get well tended to have youth with

these complex and severe conditions: three with personality disorders, two with eating disorders,

two with psychosis, one with mania. Twelve of 19 had concurrent substance use concerns as

well.

Parents in the sample were generally aware of and affected by the limitations type of MHA

concern can impose on treatment expectations. They wrote: “My daughter’s issues [personality

disorder] are proving difficult to treat…she has tried alot [sic] of treatments and therapies with

little success” (10089); “The situation with our daughter [personality disorder] will be a life

struggle” (10021); and, “It takes so long for people to get well even after they get quality care”

(10082).

There were also important contextual factors at the systemic level that could not be measured

with a survey tool but were theorized to impact both mechanisms and outcomes - conditions such

as long wait lists and a general lack of specialized services which make access to timely,

Page 110: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

97

appropriate, quality MHA care difficult even with the help of a Navigator. Another known

important contextual condition for this population was the role of privacy, consent and capacity

legislation. These three themes were hypothesized based on the background literature and data

and were notably present in the qualitative data; 41 of 71 clients (57.7%) provided over 80

quotations that spoke to these types of systemic barriers.

7.1.1.2 Theme 2: You can’t navigate to services that don’t exist

The term navigation presumes there is a system in place to navigate, but the research underlying

the FNP suggested that the system lacks not just integration between services but absolute

capacity in terms of specialized service providers for youth MHA treatment. This presents a

known significant barrier to providing effective navigation services as one cannot navigate to

services that do not exist. The FNP recognizes this constraint and strives to piece together the

best basket of services given what is available. For the clients in this sample, the results

suggested that sometimes the FNP was able to find solutions, but sometimes families were

disappointed as the solutions they found were less than ideal.

For the most part, clients of the FNP were still appreciative of their Navigators’ efforts, even

when the results were less than satisfactory. Most clients kept their comments brief: “Our

exposure to the shortcomings of the system has been a shock” (10091); “FNP did their best.

Service providers are extremely limited” (10025); “The initial intake and assessment were very

good however the options for my son were limited” (10129); and “My main problem was not

with FNP which I support wholeheartedly, but with the lack of options/resources available for

them to recommend” (10005).

Many, however, took advantage of the opportunity to express their frustration and

disappointment:

“First and foremost, the Project wasn’t the least bit helpful except to make it clear that there

was very little expeditious or effective help for Michael in the public sector…Any

improvement that [Michael] has enjoyed has come because we moved him to the private

sector ([Michael] is now in a program in the US) which is firstly, immediately available,

secondly, is comprehensive in figuring out if there is a link between addiction and mental

illness, and thirdly, has the willingness and capacity to treat these kids in a macro way. Our

system is not effective and the Project is reflective of that ineffectiveness…In treatment

addiction and mental health issues for young adults, the system is woefully inadequate. It is a

system where kids get “punted down the road.” There is no leadership or infrastructure in

Page 111: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

98

place to deal with these kids and their issues, and further, the system doesn’t seem to have

been designed contemplating that addiction and mental health issues may be related.

Addiction facilities aim to get kids “clean” and then send them back out into the world

without dealing with the possibility that there may be underlying causes of that addiction.

Mental health resources are extremely limited, very difficult to navigate and often all but

impossible to understand” (10003);

“Sadly for us, this program was no help at all. My son is now 18. I have very little hope for

his future. Our mental health system tells me, “this is normal teenaged behaviour” or “he is

overprivilaged [sic] and this with [sic] what we get.” My son grew up in a very normal house

with 2 biological parents that loved him dearly, there is no reason for his life to look like it

does. Now the criminal justice system will be his [sic] mental health advisor” (10012);

“We struggled for years trying to negotiate the system. There are huge deficiencies in the

mental health system. Sick Kids was awful, Sunnybrook was a disaster, CAMH had no

programs for youth, the school was useless, the police response was traumatic…The system

is entirely broken” (10068);

“It did not really help me or him to deal with the system. I want to be clear that this was not

[the Navigator’s] fault in any way. She did what she could, but it seems there is not much

help available in Canada (US treatment programs were recommended). The inpatient

psychiatrists at Sunnybrook seemed overwhelmed and too busy, the nurses were not

psychiatric nurses for the most part, and there is no therapy offered to the inpatients, just

drugs” (10015); and,

“Please note again I do not feel this is really the fault of Family Navigation who I believe did

try to an extent…there is just NOTHING in the Canadian Medical system to help teens with

Mental Health issues other than expensive individual therapists who may or may not be

equipped to assist. The other option we were told was to go to Sunnybrook emergency if our

child was suicidal. We have done that now twice and ended up leaving both times after a few

hours of waiting as we were triaged and not seen” (10018).

The results suggested that families in this sample were significantly struggling but that the

current system could not meet their needs. The data indicated that age, type of MHA concern,

and required service types were the most commonly associated context variables for this theme.

Clients who spoke about this overall lack of services tended to have transitional-aged youth (18

to 24 years old) for whom there are fewer targeted resources available; were often seeking care

for concurrent addiction (n=14) or complex, severe conditions like personality disorders (n=7),

bipolar disorder (n=3) that are relatively less common; and were often in need of service types

that are particularly lacking in the current system, such as residential treatment (n=6), day

treatment (n=2), and aftercare programs (n=2).

Page 112: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

99

7.1.1.3 Theme 3: Existing services lack accessibility and continuity of care

The second systemic-level contextual theme that was present in the qualitative data was related

to the inadequate supply of services, which was that the services that did exist were inaccessible,

untimely, and discontinuous. Lack of capacity has created service fragmentation and gaps in care

pathways, which made it inordinately difficult for some families in this sample to identify and

access the appropriate services in a timely manner. Twenty-four unique clients (33.8%) spoke to

the impact of this context on their experience and outcomes:

“We had to wait 6 months for her first psychiatric appt [sic] and employed therapists and

psychologist and DBT to keep her alive in the interim. That year cost us thousands and we

remortgaged but most can’t. Nothing was covered except annual max of $200 with health

insurance provider. Even now there is no coordination between psychiatrist, doctor,

endocrinologist, and therapist. The whole system is shocking and disgraceful. All the

advertising and sponsorships and walks are really just crap. The system is disjointed and

impossible to navigate. Agencies competing for funds and NOT comprehensive care. Psych

wards were basically homeless shelters. I work in education and the chances of arranging

help for kids is just as elusive” (10091).

This was a systemic barrier that also impacted the FNP’s ability to perform:

“There was certainly some frustration that is not coming across with the questions above.

There were times when practitioners were recommended/suggested, but then appointments

with them never happened because of paperwork issues between the recommended

practitioner and the regular physician’s office. This was frustrating as we often felt

encouraged by the people who were being recommended, only to go 1-2 months and nothing

would happen. Also initially I was hopeful in terms of the contact I received at the beginning,

but then it took a few months (3-4) before the first real connection to a recommended service

was made. Sometimes, at the beginning, it felt like I was back where I had been over the

previous few years, and that was frustrating” (10107).

Most clients were particularly frustrated by how long it takes to receive services across the

system. One client explained, “I spent days and weeks trying to find out what was available to

her as things got progressively worst [sic] only to find a huge waiting list;” (10049); and another,

“There’s a lot of “try this” or “maybe this” and attention is available 6 to 8 months from now”

(10003); and another, “The problem has been that we are still on wait lists for both case

management and CAMH treatment. We completed the intake process for these about 9 months

ago but we are still waiting for services to begin” (10100). Some families encountered waitlists

as long as 18 months: “We were on wait lists for any appointments or assessments and were

looking at 12-18 months. We essentially had to wait until an emerg[ency] admission bumped

Page 113: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

100

him up the list. We had to wait for criminality to access school-based interventions” (10068);

and, “The wait lists for things are also extremely long – some years…so there is no ability in

some cases to get resources (we were looking for “residential mental health” assistance)”

(10125). This was a challenging situation for some parents to manage: “I wish it didn’t take so

long to see a specialist. It’s difficult to see your child go through such depression while you’re

waiting for your appointment” (10035).

Another important issue was continuity in service. Pathways to care for MHA concerns are often

complex, involving multiple providers, and long-term, in addition to this being a particularly

difficult population to engage. Continuity of care is essential, and the FNP strives to “get in the

boat” with families for the duration of the help-seeking process because, as one parent described:

“The most challenging thing as a parent was when my son was an inpatient at Sunnybrook

and I felt helpless and afraid about what he was going through, not being able to speak

directly to the doctors regularly, and what the next steps would be after he got out” (10115).

In this sample, Navigators often stayed engaged with families over the long-term, but could not

always guarantee the services to which they referred would be as responsive: “I was more than

happy with the services of the FNP. Unfortunately, the services that were recommended to us

took months (CAMH) to get and lasted 2 hours when we got them and my son lost interest in

getting help” (10002); “He has been taken to hospital twice threatening suicide and released with

barely a question” (10012); and,

“The questions about treatment services were hard to answer, because our child has not yet

received clinical services or treatment yet. We do pay for private counselling support as a

way of bridging until we can access treatment. It is going on one year now. A great deal of

delay originated with our own G.P., whose office, twice, omitted to include [sic] critical

information in the referral to Sunnybrook” (10047).

Clients also spoke about inaccessibility of services due to such barriers as prohibitive eligibility

criteria, geographic constraints and costs: “Hard to say it in multiple choice, but we feel helpless

at times that “the system” is full of “gatekeepers” who hold the balance of power to let your child

enter, or else just let them flounder in a sea of despair” (10047). Catchment areas were a

common barrier to care: “I wanted someone from Sunnybrook to help my son as I work here, but

they wouldn’t as I am not in the cachment [sic] area” (10035); and, “It was incredibly stressful to

have to go downtown to SickKids for every appointment (and there were many). We didn’t know

where to start to look for resources in the community” (10062).

Page 114: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

101

Financial resources meant some families had the option to go elsewhere rather than wait: “We

were forced to take our child to the USA due to wait times and the lack of cohesive services

available” (10063); “I am deeply sad and frustrated about how difficult it is to obtain paid for

services through OHIP. We luckily had the financial means to pay for services, otherwise I truly

believe our daughter would not be alive today” (10049); and “I can’t imagine what other families

with less are doing” (10082). Unfortunately, other families did not have this option, as “The only

options provided were all in the US and extremely expensive,” (10018) and yet “There is no

funding available to seek quality care” (10131).

No particular C-C dyads stood out, which may reflect the widespread impact of a systemic lack

of capacity for youth MHA services. The merged dataset suggested that clients in this sample

who commented on this theme had a wide range of associated types of MHA concerns; had

youth of all ages; and were seeking a variety of service types.

7.1.1.4 Theme 4: Privacy, consent and capacity legislation

Another important theme present in the qualitative data was the unique role of privacy, consent

and capacity legislation as a barrier to care in the field of youth MHA concerns.

Sixteen clients (22.5%) referred specifically to the negative impact privacy, consent and capacity

legislation has had on their navigation and general help-seeking experiences. Most were

extremely frustrated by their exclusion from care because they felt it impeded their ability to

parent their child and that they could be an important resource in recovery:

“You are not taking into consideration the 18 year olds…Our daughter had gotten very

immature in some aspects and certainly was during her mania. Even when she was sedated

and in four point restraints in a psych ward they wouldn’t tell us anything!!!!!!” (10091);

“My experience vis-à-vis the professionals involved in my daughter’s care has been

haphazard despite the ongoing involvement with my daughter. This has been due to various

reasons including the fact that I was originally told that because she was considered an adult,

she could make her own decisions. This information I received when she had been admitted

to the hospital the week after her 16th birthday. Apparently her decisions that lead to her

hospitalization were considered sound enough to exclude me” (10105);

“Privacy laws are important for the child but severely impact what a parent can do. The

parent is usually the one who most wants to help their own child. The road blocks to assisting

one’s own child are now too much. VERY frustrating as a parent” (10008);

Page 115: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

102

“Due to the youth’s’ age, above 18, no one can obtain any information regarding any of the

treatments, diagnosis etc. unless the patient (youth) approves it. Due to this factor, I could get

no information whatsoever as she was 21 at the time. When it comes to mental health,

especially if the parent is the one seeking, finding and getting the patient to admit themselves

or participate in it, the parent should be able to gain pertinent information to assist both

parent and child to work towards the same goal” (10020).

“When a youth of 17 or 18 is in severe depression and the primary caregiver is kept outside

the doctor’s room, it upset me as a parent as I knew more about the situation than my son. I

should have been more involved in the plan to get him help” (10035);

“It is frustrating that if a child is of legal age, then the medical professionals do not share

information with parents. My son’s initial diagnosis, which I arranged, was not shared by the

assessing psychiatrist or his GP, and if it was, then I could have addressed his treatment

needs before it turned into a crisis. Now I feel that he will never get better – too little, too

late” (10061); and,

“I also think it is frustrating that when a troubled youth turns 16 he/she is suddenly allowed

to keep things confidential with the health care provider. How can we help them as parents if

we are not aware of their needs and concerns?” (10066).

The findings suggested that there may be a degree of misinformation communicated to parents in

this sample about the criteria for privacy, consent and capacity being age-based. Other clients

highlighted the fact that the same applied to parents of youth under the age of 16 as well: “It was

appalling that at 13 patient confidentiality excluded us as parents from having any information

about his diagnosis and symptoms” (10068). Regardless of age, restrictive interpretations of

current legislation were a significant barrier to care for these families, despite the fact they were

often in the primary role of interacting with the system and negotiating access to resources on the

youth’s behalf.

Under this theme, age was the most commonly associated contextual variable. While quantitative

analyses did not always find age to be a statistically significant influence on experience and/or

outcomes, qualitative data indicated that age did play a role for some families; thirteen of the 16

clients who spoke about the impact of privacy, consent and capacity legislation on their

experience and/or outcomes had youth who were aged 19 years or older. Altogether, it was

evident from clients’ feedback that the contexts in which families were seeking help for their

youth with MHA concerns were highly complex and impactful; and that understanding these

contexts is essential to appropriate service navigation and provision.

Page 116: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

103

7.1.2 Mechanisms

Qualitative data were also coded for mechanisms. The three mechanisms or service

characteristics identified by the program theory and measured quantitatively were accessibility,

continuity of care, and family involvement; these three themes were pre-set, and others were

again allowed to emerge from the qualitative data in order to add to the program theory about

which mechanisms may be operating and how; and if and how the experience of these

mechanisms was influenced by context.

Pawson and Tilley state that mechanisms naturally involve both resources and reasoning, but do

not provide a method for distinguishing between these two processes. Recent contributions to RE

suggest that the importance of distinguishing these two mechanistic aspects have been

understated to date, which has contributed to inconsistency and confusion in identifying how

mechanisms operate and how C-M-O configurations are generated (Dalkin, et al., 2015). Instead,

it has been suggested that mechanisms (M) be further categorized as related directly to the

resources or activities provided by the program (m(resource)), or to the human reasoning in

response to receiving those resources (m(reasoning)). This allows for clearer specification of

how resources and reasoning are interacting with each other and with context to generate

outcomes.

For this reason, qualitative data referring to one of the original three proposed mechanisms were

coded as m(resources) because it is the provision of these three service characteristics that were

thought to define navigation. In addition, when qualitative data referred directly to clients’

attitudes, values, feelings, or beliefs that resulted from the provision of those resources, phrases

were coded as m(reasoning) (Dalkin, et al., 2015).

7.1.2.1 M(resources)

7.1.2.1.1 Theme 1: Accessibility

Although the FNP cannot directly influence the accessibility of the services it refers to, it can

guarantee that their own Navigators are highly accessible in the meantime. The FNP provides

accessible navigation by organizing its services to respond to families’ needs, whether this means

offering clients extended and/or flexible hours of operation; having an intake line with a live

voice on the end; the ability to communicate by phone, email, Skype, or in person; free services;

Page 117: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

104

and/or guaranteed response within 48 hours. To families, these are resources that should enable

them to manage their youth’s care. Qualitative data supported the quantitative findings that the

FNP is excelling at providing highly accessible care to families of youth with MHA concerns

within and across the GTA.

Nearly 20% of clients (n=13) specifically spoke to the FNP’s level of accessibility. In general,

clients were impressed and grateful to be able to access the FNP itself whenever they required.

Clients reported, “Thanks for FNP – it was there when we needed it” (10035); “I was very

impressed with the fast response” (10029); “They were very responsive, providing resources

very quickly” (10110); “In my view, one of the best aspects of Family Navigation was how

quickly the intake procedures and involvement with the Family Navigator happened. In our case

this occurred within two weeks – a miracle!” (10037); and “Thank you Family Navigation for no

wait list and always being available to provide prompt information for my request” (10049).

Immediate access to care is particularly important in crisis situations as it can prevent both

worsening of the MHA conditions as well as inefficient and ineffective use of acute care (i.e.

emergency department) services. The results suggested that the FNP was adept at responding to

clients’ crises in this sample: “We met our Navigator when our child was in crisis. We received

quick in-take and he was seeing a psychiatrist very quickly” (10088); “My daughter was

receiving the care needed [for suicidality and self-harm] shortly afterwards” (10134); and “We

immediately had one visit and about three phone calls with FNP including [Jane] meeting

[Medical Director]” (10097).

7.1.2.1.2 Theme 2: Continuity of care

As previously mentioned, the FNP aims to “get in the boat” with families and remain engaged

throughout their help-seeking journey. In this sample, it was a common complaint that the

existing service system is extremely discontinuous, and clients were highly aware of the FNP’s

efforts to remain available to the family as long as required. Twelve clients (16.9%) explicitly

commented on the ongoing involvement of their Navigators and the frequent follow ups they

received. Overall, the twelve clients who spoke to continuity of care had been engaged with the

program for longer than the sample mean (28.2 weeks vs. 24.0 weeks). Clients commented:

“Very good at keeping in touch and trying to find solutions to our problems” (10103); “They did

not abandon us and stayed connect [sic] when initial steps did not work and we were able to try a

Page 118: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

105

different option which did help” (10113); “The Family Navigation [sic] continuously checked in

via email to see how things were going” (10001); “I’m very impressed with…the frequent follow

up” (10029); “She [Navigator] remained in contact with follow up phone calls after the therapy

sessions started” (10087); “Support staff kept emailing me on and off just to see if we had found

him and also to make sure I was ok” (10124); and,

“In the beginning, we were in contact with our navigation [sic] often. Then my son left to

live in another country for 6 mos. When we had problems again, when he came back, we

contacted our navigator again. We are satisfied with their follow up, including the parent

advocate. Thank you for the help and follow up” (10092).

The FNP also strives to facilitate continuity of care between the providers to which it refers. This

is more difficult as they cannot control the service system, but the data suggested they were

successful in facilitating some cases: “I was also impressed with how the Family Navigator was

able to quickly and directly liaise with our son’s former therapist at CAMH as well as a CAMH

psychiatrist who had conducted an assessment” (10037); “They are always available to help me

find services for my daughter and have even gone further to help when she was planning to

attend post-secondary schooling outside the area” (10049); and “Once [John]’s condition was

stabilized, they also made the referral to an experienced psychiatrist to monitor [John]’s ongoing

mental health” (10009).

7.1.2.1.3 Theme 3: Knowledge, insight and expertise in youth mental health and addictions

The results of the thematic analysis suggested that the specific knowledge, insight and/or

expertise in the youth MHA system offered to families was a standalone mechanism in and of

itself; of all program resources and activities, clients most often referred to the value of

Navigators’ knowledge, insight and expertise (n=23, or 32.3% of clients providing qualitative

feedback).

Some clients spoke specifically to the Navigators’ knowledge, insight and expertise on the nature

of youth MHA concerns themselves, and how families could better manage as a result. One

client explained, “The Family Navigation Project allowed my situation to be finally “heard” by a

team of professionals who collectively provided suggestions as to where I might get help and

explained how they might help” (10113). Others commented, “Their insight into [David]’s

condition and suggestions as to treatment were invaluable” (10009); “Your service was

Page 119: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

106

invaluable and worked as a mediator between my child (who was 21 then) and I as I seemed to

be where her finger was pointing and I could not understand how or why” (10020); and “I did

and still do appreciate all of the ideas, information and support from the navigator” (10081).

Other clients spoke to the Navigators’ knowledge, insight and expertise related to the service

system and providers available to this population: “The Family Navigation Project was great at

referring to services that I either did not know about or was having trouble accessing” (10100);

“Given the availability of so much information online, I wasn’t sure I would get much value but

the Navigator definitely provided new options” (10041); “The lists that FNP has are constantly

being updated which is really really helpful” (10021); and “We would never have found this

service on our own” (10037).

A few more clients spoke specifically to the Navigators’ ability to use their knowledge, insight

and expertise to make highly appropriate matches between families’ needs and service providers:

“She found the perfect therapist for my daughter who in turn recommended a treatment facility

that was instrumental in helping” (10094); “…guided us to exactly the help we needed at the

time” (10053); “She has provided so many resources that are excellent” (10127); and, “The

challenges I faced and still face shook me. [Navigator]’s support, problem solving discussions

and access to professionals with special background in areas I needed to access made

considerable difference” (10097).

7.1.2.1.4 Theme 4: Family support

Family support was theme that was assumed to fall under family involvement based on the

definition employed. Instead, as with the previous theme of knowledge, insight and expertise, the

qualitative results suggested that family support should be considered a standalone mechanism.

Fourteen clients (nearly 20.0%) specifically noted the support provided by the FNP. Clients said,

“[Medical Director] and his team supported our family as we looked for appropriate services for

[James]” (10009); “The family navigate [sic] that I spoke to was kind and caring and very

supportive at a time when I felt very alone” (10062); “Thanks as well for such a calm and

sympathetic ear” (10082); “[Navigator] took so much time to comfort, reassure, and encourage

me in the early days of the diagnosis” (10099); and “I really appreciate their support and

professionalism and caring” (10134).

Page 120: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

107

7.1.2.2 M(reasoning)

From the background research, program theory and primary qualitative data results on

m(resources) above, it was clear that having continuous access to expertise and support helped

families. As mentioned previously, qualitative data was also unique in its ability to explore how

these resources impacted families. In general, RE theory proposes that programs primarily

achieve their goals by providing resources that directly influence clients’ reasoning in terms of

their attitudes, values, feelings, and/or beliefs. This reasoning, in turn, influences decisions they

make with regard to care planning for their family going forward (Dalkin, et al., 2015; Pawson &

Tilley, 1997).

Qualitative data were therefore coded for phrases expressing clients’ attitudes, values, feelings

and/or beliefs as a result of resources received from the FNP. Since reasoning is a highly

individual process, themes in reasoning were not presupposed and instead allowed to emerge

organically from the data. Several reasoning processes were evident. The first was that as a result

of the services provided by the FNP, clients felt a sense of relief; second, clients felt reassured;

and third, clients felt hopeful for the future.

7.1.2.2.1 Theme 1: Relief

Many clients spoke about the sense of relief they felt as a direct result of engaging with the FNP

(n=18, or 25.4%). Relief occurred on two levels. The first revolved around the physical relief

from the burden of seeking help for youth MHA concerns. It is understood that the system is

extremely complex and difficult to navigate, requiring an inordinate amount of time and effort to

find, understand and access the range of available resources. This was an incredibly stressful

situation for families who are already dealing with stressful youth MHA concerns and having a

Navigator available provided a huge sense of relief by helping to reduce the overall time and

effort clients had to invest themselves. One client explained:

“When the navigator told me that she would help me make calls and find services because

she appreciated that it was a challenge and she acknowledged that I was working and caring

for my daughter and family, I felt incredible relief. I cannot tell you how much I appreciated

having a partner to navigate the system” (10062).

Six other clients expressed similar sentiments, such as: “I know I can work with FN [sic] to find

new options and this has taken some of the pressure off me to constantly come up with new

Page 121: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

108

treatment programs” (10021); “I am exhausted by the whole process, but so appreciate

everything the Navigation project has done to help” (10061); “Such timely access greatly

reduced stress and anxiety” (10037); “Navigator definitely…cut down on the time and

uncertainty of searching” (10041); and “Having a service such as yours helped me to look at

other possibilities that I could consider for my child during a very stressful time for all

concerned” (10058).

Clients also spoke of the emotional relief associated with finding a service provider who cared,

listened, and most importantly, understood. Twice the number of clients referred to this

emotional sense of relief over being heard (n=14). One client expressed this succinctly saying,

“You have no idea how important having someone who understands what its [sic] like at the

other end of the line” (10082). Another explained, “It gave me, the sole parent who was alone

and who had no one who really understood our situation, support from a group that really “got”

our case” (10113).

Other clients echoed: “I appreciate that I have been heard by a Family Navigation staff person

and she was kind” (10018); “We were grateful to have another ear to hear our concerns”

(10102); “Your service provide [sic] guidance and as well an outlet for me to let off steam due to

my frustration in not being able to get the help my daughter needed in a timely manner” (10058);

“I thought it was the sweetest thing when after my son ran away, your support staff kept emailing

me on and off just to see if we had found him and also to make sure I was ok. Such a caring

group of people. Thank you” (10124); “Family navigation offered me a piece of mind [sic] when

I had nowhere else to connect” (10090); and “She [Navigator] was very concerned about me and

how I was handling the stress, and the impact on family dynamics and marriage. I will be forever

grateful for her kind and gentle voice on the phone” (10099).

7.1.2.2.2 Theme 2: Reassurance

Families of youth with MHA concerns face many challenges to their ability to effectively parent

and seek help for their children: stigma, scarce resources, unclear legislation, and symptoms that

affect the entire family. The FNP outwardly acknowledges the complexity and stress that

families can face when trying to effectively and efficiently manage their child’s care, and

purports to help by offering continuous access to support and comprehensive, objective

information about treatment options. Thematic analysis of the qualitative data suggested that one

Page 122: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

109

of the reasons this information is, in fact, helpful is because it helps to reassure clients that they

are doing right by their child; parents (and other caregivers) in this sample wanted to know they

were doing the best they could for their youth with what they had available to them. This

involved reassurance that parents themselves were doing all they could as parents to help their

children, and/or that the professionals or resources they enlisted to help their children were the

most appropriate available.

Ten clients (14.1%) spoke to this topic. They explained how the FNP reassured them they were

doing their child justice, despite the complex system:

“The service we received was very useful because we wanted to know if there were any other

options we may have missed. It turned out that there weren’t but that gave us the confidence

to know we are on the right track and doing all we can for him” (10038); and,

“I think your programme is an important bridge to mitigate some of the confusion, fear and

helplessness that a parent can feel when knowing where to turn and how to access

appropriate services for their child, and also helping to discern if a parent’s concerns are

valid or not and in a non-judgmental way” (10067).

Other clients reiterated: “FNP was unable to add much to what I already know, but provided

reassurance that I was pursuing the best avenues” (10025); and “It was useful to talk to our

Navigator to understand the landscape of services available. As a parent, I wanted to make sure

that our family reasonably explored all services available” (10010).

Clients also mentioned how the FNP created confidence in the treatment providers selected:

“They solidified my decision” (10134); “We feel that the worker was instrumental in leading us

to the right resources and giving us great advice” (10016); “It has given us more choice which

results in being able to confidently tailor a selection to something that fits our daughter the best it

can” (10021); and “[Navigator] further enhanced our confidence in the supports that were being

recommended to us” (10037).

7.1.2.2.3 Theme 3: Hopefulness

The results suggested that as clients experienced relief and reassurance through their engagement

with the FNP, they also gained hopefulness for the future. Although they recognized that many

MHA concerns would be chronic and/or recurrent, clients of the FNP felt hopeful that they

Page 123: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

110

would be better able to manage their youth and family as the situation evolves as a direct result

of their engagement with the program. Nine clients spoke specifically to this topic.

Clients explained: “It looks at the entire problem and the people involved and helps on all fronts.

One is not left at a dead end when initial steps may not work. It gives hope” (10113); “I believe I

am better prepared as a parent to deal with the problems as they arise. Knowing that I can contact

my navigator for advice definitely helps” (10087); “It has improved our ability to cope with our

son’s illness so much better” (10127); “I do feel now that I have somewhere to turn if things go

off the rails and I’m very thankful for that” (10081); “It does not always work, but at least I

know we can continue to try” (10021); “This gave me a real boost and improved my outlook and

hope for the situation” (10100); and “We remain optimistic and are hopeful that the navigation

project will continue to act as a resource as our situation evolves and changes” (10102).

It was interesting to reflect back on the theory-building discussions with the program team

around what they hoped to achieve (refer back to Chapter 2 and/or Figure 1, Appendix B).

During this process, team members specifically mentioned wanting to offer families “a sense of

relief,” to “encourage confidence” (i.e. by providing reassurance) and “hope for the future.” For

most of the 71 families who provided qualitative data, the results suggested the FNP was

operating as intended.

7.1.2.3 The link between resources and reasoning

According to the program theory, mechanisms occurred in unique combinations for unique

clients; generally, clients were expected to experience all mechanisms to varying degrees

according to their needs. Moreover, earlier thematic analyses suggested that several different

reasoning responses occurred as a direct result of those mechanism resources. This, in addition to

the fact that each client was coded at a number of nodes and allowed to make a number of

references, yielded a wide range of M(resource)-M(reasoning) dyads that contributed to

understanding how navigation helped families. The commonly recorded dyads most often related

to a sense of relief as a result of the various resources offered; the single most common was that

of expertise and relief (n=10), followed closely by links between support and relief (n=9), and

continuity and relief (n=7). Combinations are presented in Table 32 below where “N”

corresponds to the number of unique clients coded on both mechanisms in the specified dyad.

Page 124: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

111

Table 32. M-M dyad combination by theme

M-M dyads

M(reasoning)

Relief Reassurance Hope

M(resource)

Accessibility 5 3 2

Continuity 7 1 4

Expertise 10 3 6

Support 9 2 4

7.1.3 Outcomes

These theory-building discussions around what the FNP hoped to achieve led to the selection of

a set of quantifiable, standardized outcome measures that could theoretically be expected to

result from the services provided. For example, the FNP team wanted to “encourage

confidence,” and so empowerment and a corresponding standardized outcome measure – the

FES – was selected. In place of “a sense of relief” and/or “hope for the future,” the FNP team

settled on an improvement – any improvement – in family quality of life, quantified using a

modified version of the BCFQoLS. Standardized measures such as these were essential because

they allowed for quantitative analyses and comparison. However, other than service satisfaction,

the selected outcome measures were distal, highly complex and subject to a wide range of

influences. Particularly with small sample sizes and cross-sectional designs, statistical

significance is difficult to detect and effect sizes are likely to be quite modest (Tabachnick &

Fidell, 2007). Moreover, in a program providing unique combinations of services to unique client

contexts, unique outcomes should also be expected.

Clients’ qualitative feedback was therefore coded for phrases that spoke directly to a result of

being engaged with the FNP, and analyzed in terms of how these outcomes were influenced by

context and/or mechanisms. Outcomes were then simply categorized as positive or not positive.

Page 125: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

112

Once all outcomes were coded, an initial word frequency query was run to generate an

exploratory word cloud, which is a simple visual depiction of the most common words in the

coding category (Figure 2). The most frequently occurring stem word by a count of 50 was

“help” (and its derivatives: helps, helped, helping, helpful.

Figure 2. Word cloud depicting word frequency in all phrases coded as an outcome

7.1.3.1 Positive outcomes

Thirty-eight unique clients (53.5% of those providing qualitative feedback) spoke to a general

outcome of the FNP. Of those references, 31 (or 81.6%) were positive, and no contextual factors

were identified as more often associated with positive feedback than others, which lent support

to the quantitative findings that most clients in this sample were satisfied with their navigation

experience. As mentioned above, positive feedback generally fell under two themes. The first

was when services were successfully accessed and youth and family are now better off as a

result. As one client explained:

“The Family Navigation Project was a life saver. I truly believe that it was the first step that

helped to get a young man who had withdrawn from the world back out and making his way

into a productive happy individual. I believe that many of our other healthcare services

should follow this format” (10113).

Others echoed: “I feel that through involvement in the Family Navigation Project, we were

directed to a program that is working well with our son as well as the whole family” (10037); “I

did get great service which helped my son” (10035); “Not only were we directed to get help for

my son but for me as well and within a few months my son started to come back to the world”

(10120); “It DID result in their help in finding a psychiatrist for our daughter… who has been a

Page 126: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

113

godsend” (10091); “I have had such an amazing experience with family navigation services. My

son currently is thriving” (10108); “I was extremely satisfied with the family navigation project

for my nephew. He received the immediate care he needed. Family navigation connected him

with a doctor who he has been in touch with regularly” (10090); and,

“We have been thrilled with the care and with the referral to the addiction counselor. We

firmly believe that our child has the best care – just need him to continue on his path to

wellness before we can check all those boxes as ‘extremely satisfied’” (10088).

Qualitative data effectively complemented the quantitative satisfaction measures, with feedback

speaking to satisfaction with the FNP and satisfaction with the services to which they were

referred. Clients spoke to FNP-specific satisfaction outcomes above and beyond overall

satisfaction, including likelihood to recommend and support for scaling the model, a long-term

stated goal of the FNP. Results aligned with the findings from the quantitative analysis, which

suggested that overall, most clients were highly satisfied.

Clients volunteered: “I recommend this service on a regular basis to friends, patients, colleagues”

(10072); “This program needs to be located in every region. Families need support now more

than ever to get help” (10046); “I have already recommended this service to a number of people”

(10039); “We believe that every family with a mentally ill family member could use the

resources and guidance offered by this team. Something like it should be offered everywhere and

to all ages” (10009); “I would highly recommend this service!” (10094); “I have really nothing

to say to improve your services. I would like to say just keep doing what you are doing” (10124);

and “The services provided through the project were phenomenal…other cities should have such

navigation resources” (10097)

The last theme under which positive feedback fell was a more subjective sense of appreciation

and gratitude for the FNP team. The parents and youth in this sample represented a unique and

vulnerable population who had faced repeated barriers to accessing the care in the past. As a

result, when prompted for concluding feedback on their experiences, many clients used the

opportunity to simply express their appreciation and gratitude: “I am exceedingly grateful”

(10097); “I am extremely grateful that someone I knew had put me in touch with the Family

Navigation Project” (10120); “I want to thank Family Navigation for providing names of

services and people to help us through this difficult time” (10049); “I went on sheer faith in your

recommendations and grateful for the services. I have donated to your service and would be

Page 127: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

114

pleased to help on your Board of Directors or with other parents” (10065); “I will be forever

grateful for her kind and gentle voice on the phone…we are very thankful to our navigator”

(10099); “I can’t say enough good things about the program. [Navigator] has been so helpful.

The services are a godsend” (10127); “Thanks for FNP – it was there when we needed it”

(10035); “Your service was wonderful and our contact was amazing. Thank you” (10020); “I am

grateful that I found this program” (10039); “Thank you for all your help and hard work. You are

the ONLY organization that made a connection for my nephew to obtain help…Great Work!”

(10071); and, “This is the best resource and support for families with children with mental health

issues…such a caring group of people…Thank you for helping us” (10124).

7.1.3.2 Negative outcomes

It is important to note that not all outcomes were positive; and that all mechanisms can have

unintended consequences. There were also cases in which, despite Navigators’ efforts, families’

needs and expectations were unmet. For most clients, this was not the case: only seven clients

reported negative outcomes, which was 18.4% of clients providing qualitative outcome data and

9.9% of the entire qualitative population. Recalling earlier discussions about the many

challenging contexts this population must navigate, it is not unforeseen that some families will

fail to find the full range of supports they require. As one client expressed,

“I don’t mean to slam your project. I was very excited when I learned of it. I thought we

could find help here. We have…seen absolutely no improvement in our son. I am personally

falling apart over the loss of my son” (10012).

Others reported similar outcomes: “We are still weighing our options for treatment and our child

continues to be very ill and unhappy” (10018); “My son still has reoccurring mental issues my

family and I have to deal with now” (10087); “Lots of good recommendations… unfortunately,

none have really helped (10098); “When I think about the lack of services being implemented

thus far I feel discouraged” (10100); “The provider did not align with my son and so he ended up

with no care until I changed his family doctor” (10129); and succinctly linking a non-positive

outcome with low service satisfaction, “Navigation services didn’t help her, so I can’t say I’m

very satisfied with the services” (10089).

Qualitative outcomes were subcategorized simply as positive and negative, and since 31 of 38

unique clients with unique contexts provided positive feedback, it was instead the link between

Page 128: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

115

context and negative outcomes that stood out. In most cases, number and type of MHA concerns

were again the most common contextual factors associated with negative outcomes in this

qualitative sample. The merged dataset indicated that the seven clients who reported negative

outcomes had a higher number of mental health concerns overall (2.7 vs. 2.3 in the total client

population). In addition, four of the seven had concurrent addiction concerns, which the

quantitative results suggested was related to poorer outcomes; and the three with concurrent

addiction concerns had complex, chronic-severe conditions including personality disorders,

OCD, and autism. While the particular type of mental health concern was not found to be

significantly predictive of negative outcomes in inferential modelling, the qualitative data

suggested that type of mental health concern did influence, to some extent, the experience and

outcomes of navigation for several families in the current sample.

Page 129: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

116

Page 130: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

117

Chapter 8 Discussion

This concluding chapter first discusses the merged results as they relate to the research questions

of this study. Then, key findings are discussed in relation to and used to refine the original

program theory and conceptual framework for family navigation. The chapter and thesis closes

with a discussion of study limitations, strengths, implications for and contributions to the field.

Results as per the research questions, conceptual framework and program theory

This study was guided by a conceptual framework and three broad research questions, the

answers to which would describe, in depth, the clients and youth of the FNP; the impact of

family navigation on families in terms of their level of family empowerment, family quality of

life, and satisfaction with the services they received; and the influential role of context. The

results were then used to refine the original conceptual framework and program theory. It is

noted that the results reported in this study cannot be generalized beyond this sample of

convenience as both the clientele and the program have evolved since the current study was

initiated.

8.1 Research question 1

The first research question asked, “Who is the Family Navigation project serving? Is the Family

Navigation Project reaching its target population? Overall, are families satisfied with the

services they received?”

The FNP has several service mandates that help to define its target population. Broadly, the FNP

is intended to serve families of youth aged 13 to 26 with mental health and/or addiction

concerns. First, the FNP was an initiative that was started by parents, for parents. The results

indicated that over 90.0% of the sample were parents seeking help for their youth. In this respect,

the study results suggest FNP reached its clientele in this sample. When the FNP was first

designed, lived experience and extensive research by the program team highlighted two priority

sub-populations most in need of navigation: youth with concurrent mental health and addiction

concerns, and transitional-aged youth. The addition of an addiction concern(s) to mental health

Page 131: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

118

concern(s) further complicates an already challenging mental health service context. Mental

health services and addiction services are generally provided by separate entities and sectors;

often, qualifying criteria for a service in one category precludes access to services in the other

category (CIHI, 2013; MHCC, 2012; MOHLTC, 2011; Pearson, et al., 2013). Transitional-aged

youth have to navigate an additional barrier: the transition between child and adult mental health

service systems, which are also provided by different sectors (MHCC, 2015).

The sample described in this study is consistent with the populations that the FNP is intending to

reach. Families in this sample reported a very wide range of mental health concerns, from

relatively more common and manageable conditions like anxiety and depression to complex,

chronic, severe conditions like personality and eating disorders, OCD, and bipolar disorder.

Addiction concerns ranged from cannabis and alcohol concerns (most commonly), to stimulant

and opioid use, to behavioural addictions like video games. Moreover, 45.5% reported

concurrent mental health and addiction concerns. This is consistent with the rates of concurrent

mental health and addiction concerns reported in the literature; rates of concurrent addiction

among adults with a mental disorder (and youth with major depression) typically vary from

20.0% to 50.0%, which suggests the FNP has effectively reached this population (CMHA, 2013;

Centre for Behavioral Health Statistics and Quality, 2015).

Correlational results suggest extensive interaction within and between conditions, with clients

reporting an average of 2.3 and 1.7 mental health and addiction concerns, respectively. The

associations found in this study sample tended to follow established patterns in the literature

(Kessler, et al., 2005, 2012; Nguyen, Fournier, Bergeron, Roberge, & Barrette, 2005; Pearson, et

al., 2013). For example, mental health and addiction concerns were related to both age and

gender in this sample, with older males tending toward substance use and complex mental

illnesses, whereas younger females tended toward anxiety, self-harm, eating and personality

disorders. Some concerns themselves were also highly associated and reflect the literature base,

such as the links between depression and anxiety, suicidality and self-harm, ADD/ADHD and

addiction concerns, and eating and personality disorders.

Consistent with a wide range of interactive mental health and addiction concerns, clients in this

sample had complex case histories. Nearly all had previously received some form of youth

mental health or addiction service; over a third had previous ED visits or inpatient stays; and

Page 132: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

119

over a quarter had some history of legal involvement in their care. The high levels of ED and

inpatient use among this sample reflect the increasing trend data for ED and inpatient use for

youth mental health reasons that was recently released by CIHI (2017).

A wide range of reasons were reported for seeking navigation, from general guidance to specific

service requests. Most often, families were seeking recommendations. Since nearly all families

had already connected with the service system, it is possible that they continued to have unmet

needs and/or were dissatisfied with the recommendations given. They were also frequently

simply seeking information. This aligns with the focus group data from which the FNP was

conceived; families in the focus group reported that simply having access to comprehensive,

objective information about their treatment options was immeasurably helpful (Appendix A).

Although sample size was relatively small and this may threaten generalizability of study results,

the specific reasons clients had for contacting the FNP may reflect known gaps in preventative

community, outpatient and step-down services (CMHO, 2016a). For example, clients were

seeking services such as crisis supports, access to day treatment, and aftercare. Identifying and

addressing such gaps could have positive system-wide effects.

A number of clients were specifically seeking information on or a referral to residential

treatment. The FNP purposely developed an aptitude for navigation to residential treatment due

to the fact that this is a known, significant gap in Canadian mental health and addiction treatment

(CMHO, 2016b). There are very limited choices of residential treatment centres for youth with

mental health and/or addiction concerns in Canada; there are only a handful of centres across the

country that are typically have extremely long wait lists and restrictive out-of-pocket costs.

Moreover, mental health and addiction treatments are generally not provided concurrently, and it

can be difficult to find a residential program that will treat both concerns despite the high rate of

co-occurrence. The results of this study suggest that residential treatment services in Canada are

still inaccessible for families of youth with MHA concerns.

The FNP is an organization positioned to overcome several common barriers to accessing care.

The first barrier for many help-seeking families is geographical in that public services are

typically restricted by catchment area. However, because the FNP is privately funded, it is not

subject to the same catchment and eligibility restrictions as public services, and as such intended

to provide services to the whole of the GTA. The results of this study suggest the FNP has

Page 133: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

120

extensive reach, with clients distributed fairly evenly across the SHSC catchment area, the City

of Toronto, and the GTA. The second mandate is to provide equitable navigation services at no

cost to families; and to recommend services that are considerate of families’ financial resources.

Clients in this sample were predominantly Caucasian and of high socioeconomic status, which is

certainly not reflective of general demographic trends among those with MHA concerns. These

results were presented to and discussed with the program team, who were able to further

contextualize the finding. The program team explained that the statistics are likely reflective of

the fact that they do not advertise their services, and that in the early stages of operation, clients

arrived at the FNP primarily through word of mouth and social networks that included members

of the FNP’s Parent Advisory Committee, who were predominantly Caucasian and of high

socioeconomic status. The program team also noted that these findings are not reflective of their

current clientele, which has grown considerably in size and variability as program reach

continuously expands. In addition, this study employed a convenience sample and it may have

been the case that families with higher socioeconomic status had contexts (such as time and/or

willingness) that better enabled them to respond to the survey.

Although the results of this study were taken from a small and non-representative convenience

sample which may not reflect all families with youth MHA concerns, they are consistent with the

determination that the FNP is reaching its target population by providing navigation services to

families of youth and young adults with mental health and/or addiction concerns.

With regard to whether or not families are satisfied with the services they received, study data

supports the conclusion that overall, clients in this particular sample were highly satisfied with

the services they received. This included both satisfaction with the FNP itself, as well as

satisfaction with the services to which the FNP referred families. Scores for satisfaction were

significantly higher than for any other outcome, which is thought to reflect both the considerably

more proximal nature of the outcome and the appropriateness of the theorized mechanisms.

8.2 Research question 2

The second research question asked, “Do families perceive the Family Navigation Project to be

providing accessible, continuous, family-inclusive care? How does context influence perceived

experience of the program?”

Page 134: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

121

Study results suggested that the FNP provided care that most families in this sample perceived as

accessible, continuous, and family-inclusive. The results of the quantitative analyses indicated

that item-specific and total scores across all three mechanism measures were very high and the

majority of the sample was very satisfied. Several low scoring outliers on continuity and family

involvement were explained by context in the qualitative analysis. Qualitative results indicated

that most clients who rated continuity of care and family involvement items poorly tended to

either not require continuity of care or family involvement, which was typical of clients who

only wanted information; or they did not objectively have the opportunity to experience or

receive continuity of care and family involvement due to the duration of time they had been

registered with the program. This was confirmed by the quantitative results, which suggested that

a client’s number of contacts with a navigator was understandably and significantly associated

with their rating of accessibility, continuity and family involvement.

The FNP was designed to provide families with needs-based help despite varying contextual

conditions by offering them highly accessible, continuous navigation that specifically prioritizes

their involvement. If it is true that these three service mechanisms – accessibility, continuity of

care, and family involvement – are what enable them to overcome contextual barriers, then very

few contextual factors should have mattered to the FNP’s ability to navigate for families. With

regard to which factors influenced experience, the results of this study suggested the answer is

that there are a few individual-level contextual factors matter, and that there remain significant

systemic-level contextual barriers.

At the individual-level, few contextual factors were associated with perceived experience of

family navigation. Accessibility, continuity and family involvement did not appear to be

associated with typically influential factors such as demographics and type(s) of MHA

concern(s). This suggested the FNP was effectively adapting its service mechanisms to the

contexts of families in this study sample. Only two contextual factors stood out as significantly

related to the mechanisms. Suicidality concerns positively associated with all three, whereas

OCD concerns were negatively associated. The positive association with suicidality is

particularly interesting and is thought to reflect the FNP’s expertise in responding to this

particular population, in which the reported rate of suicidality was significant (17.9%). The FNP

has become particularly adept at educating families around how to handle acute youth mental

health and addiction concerns and crises, and in navigating families to youth crisis supports;

Page 135: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

122

these were some of the most critical needs and difficult situations to manage that families

identified when the program was first conceived. On the other hand, OCD is a medically

complex, chronic and relatively less common condition for which highly specialized services are

required and while some do exist (two adult-only centres are available in Toronto), access is very

limited and hindered by the need for a physician referral and long wait times (Richter, 2017). It

could be the case that there are systemic barriers to OCD treatment in particular that the FNP is

simply unable to overcome.

The results of the qualitative analysis align with research underlying the program which suggests

that there are significant systemic-level contexts that influence families’ experience of

accessibility, continuity and family involvement. These were discussed at length in the

qualitative results chapter but to review, they include an overall inadequate supply versus the

demand for youth mental health and addiction services, especially community-based step-up and

step-down services; a particular dearth in specialized services for concurrent conditions and

transitional-aged youth; poor accessibility and continuity of care within existing services across

the system; and restrictive interpretations of privacy and capacity legislation that can impede a

family’s ability to participate in their youth’s care. Overall however, the results of this study

suggest that families perceived the care they received from the FNP as accessible, continuous,

and family-inclusive.

8.3 Research question 3

The third research question asked, “Do families who perceive the Family Navigation Project as

accessible, continuous and family-inclusive experience better outcomes in terms of family

empowerment, family quality of life, and service satisfaction? How does context influence these

outcomes?”

Study results indicated that families in this sample who perceived the FNP to have provided

accessible, continuous and family-inclusive care reported significantly better outcomes in terms

of family empowerment, FQOL, and service satisfaction.

With regard to family empowerment, results suggested the impact of perceived experience was

significant but modest in effect; this was particularly true for family empowerment at the service-

seeking level. Conceptually, family empowerment is a broad concept that is likely influenced by

Page 136: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

123

a wide variety of interactive factors, and a distal outcome of navigation when compared to

navigation satisfaction, for example. This was reflected in the quantitative results by much more

modest correlations between mechanisms and family empowerment and quality of life outcomes

than between mechanisms and satisfaction outcomes. Nonetheless, the results of the inferential

analyses indicated that more positive perceptions of navigation, as defined by accessibility,

continuity of care and family involvement, predicted higher levels of family empowerment in the

home and in service-seeking for the families in this sample.

In terms of how context influences family empowerment, correlational analyses suggested that

age mattered, as did type(s) of MHA concern(s). Families with younger children were more

likely to be empowered in both domains, possibly because they are more in control of their

youth’s care at this age. Families with concurrent mental health and addiction concerns had

lower associated family empowerment scores in both domains. In contrast, families with

concerns about suicidality were more likely to be empowered. This may relate to several factors,

some of which have been mentioned previously. The first is that the FNP is thought to be

particularly skilled in managing cases with suicidality concerns; and suicidality concerns are also

extremely acute, families with this type of concern are likely to have received immediate

attention and consequently experienced a relatively immediate result or change in state. It is

possible that families were significantly empowered by this immediate response, as well as the

provision of specifically appropriate information and supports.

With regard to FQOL, study results suggested the impact of perceived experience was again

significant but similarly modest in effect size to the influence on family empowerment. The

results suggested that families who had more positive experiences of navigation were likely to

score significantly higher on FQOL. Similar to empowerment, suicidality predicted better

outcomes, likely for the same reasons detailed above. In contrast, again as with empowerment,

concurrent concerns significantly predicted poorer FQOL outcomes; this was true of families

who reported personality disorder concerns as well. This could be related to the inaccessibility of

resources for both these conditions.

In addition to the aforementioned individual-level contexts, the systemic-level contexts

determined to influence mechanisms were also expected to influence outcomes through their

interaction with mechanisms. That is, if these systemic-level barriers to care resulted in families

Page 137: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

124

experiencing poor accessibility, continuity and family involvement in youth MHA treatment

planning and management, it is reasonable to expect that as a consequence of not receiving the

high quality care they needed, they would feel considerably less empowered, particularly

because their youth’s health concerns had the potential to escalate without proper care and

support. For families with less experience of negative systemic influences and/or more

experience of positive influences, FQOL could reasonably be expected to be lower, as a result.

Qualitative data from the current sample lent support to the idea that these contexts have

significant impacts on families’ empowerment and quality of life. As a result of systemic barriers

to care, families in this sample spoke of how “difficult” and “elusive” it was to find the help they

needed, and that as a result, they felt “discouraged,” “deeply sad,” “frustrated,” and “helpless.”

Overall though, the findings from this study support the idea that navigation has a significant, if

only modest, positive impact on family empowerment and family quality of life. Qualitative data

supported the impact of family navigation on family empowerment and family quality of life as

well. Families in the qualitative sample spoke extensively about feeling relieved, reassured, and

better able to manage in their daily lives as a direct result of the FNP’s involvement. They also

felt more hopeful for the future. Certainly, this can be considered a relative improvement in

FQOL for the families in this sample.

With regard to service satisfaction, inferential modelling results suggested that families who

perceived the FNP as providing accessible, continuous, family-inclusive care were more likely to

be highly satisfied with the services they received. Naturally, the impact of perceived experience

was exponentially greater on satisfaction with the FNP itself, again reflecting the proximity of

the outcome and the appropriateness of the measures. Overall model fit was substantially better

for this dependent variable (i.e. NAVSAT total score) as well. Another indication that the

mechanisms were operating as theorized is that only one of seven contextual variables identified

as significantly associated with the outcome was a statistically significant predictors of

individual differences in NAVSAT total score. While legal involvement was a barrier that was

found to be a significant negative predictor of satisfaction with navigation, more important is the

finding that satisfaction with navigation was not significantly influenced by factors like youth’s

improvement, number and type of MHA concerns, and reasons for needing navigation. In this

study, factors that were expected to be negative influencers of outcomes were not impactful. This

may suggest that the provision of navigation that is perceived as highly accessible, continuous,

Page 138: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

125

and family-inclusive may mitigate, to some extent, the negative impact of those factors such that

families’ satisfaction with the navigation services themselves is preserved.

With regard to satisfaction with referred services, the influence of context was similarly limited.

Modelling results implied that clients were more likely to be satisfied with the referred service if

they were specifically seeking a psychiatrist. Youth psychiatrists are a limited and difficult to

access resource, and finding the right patient-doctor fit is particularly important to engage this

population (CMHO, 2016; Chovil, 2009). That the need for a psychiatrist was associated with

high satisfaction may be a reflection of the FNP’s ability to make appropriately individualized

assessments and facilitate timely access to the right psychiatrist for that youth and family’s

needs.

Again, the same systemic contexts influencing mechanisms and the other outcomes mattered for

satisfaction too. First, they mattered because they highlighted the innovation and quality of the

FNP’s services. The qualitative study results indicated that mental health and addiction services

were often inaccessible for families seeking care on behalf of their youth. In comparison, clients

valued with level of accessibility, continuity and involvement in their navigation services. They

spoke of being “thrilled with,” and “so impressed by” the fast intake and ongoing responsiveness

relative to their previous experiences; and of “finally being heard.” In contrast, clients who

expressed dissatisfaction in the qualitative sample did so not so much because of the navigation

services themselves, but because they were unable to access the services to which they were

referred, or because the service did not end up being helpful.

Overall, the study results strongly supported the conclusions that most families in this sample

were satisfied with the services they received; that perception of navigation significantly,

positively impacted satisfaction; and that context often mattered but to varying extents.

8.4 Results in relation to the conceptual framework and program theory

As a whole, the results of this study supported the original program theory that the program is

situated amid a variety of influential individual- and systemic-level contexts, but that with

appropriate inputs, Navigators are enabled to provide accessible, continuous, family-inclusive

care to families of youth with mental health and/or addiction concerns, which has a significant

Page 139: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

126

positive impact on their level of family empowerment, family quality of life, and satisfaction

with the services they receive. This pathway was particularly true for satisfaction; a comparison

of quantitative model fit statistics suggested satisfaction with navigation was the most proximal

outcome to the proposed mechanisms; and that the proposed mechanisms had the largest

combined main effect on satisfaction with navigation compared to any other outcome.

In view of the theoretical emphasis on individual and system-level contexts presented in the

original logic model (Figure 1, Appendix B), the results of this study strongly supported the

importance of understanding the influence of context on families’ help-seeking experiences.

Quantitative results highlighted extensive associations amongst context variables themselves,

and between context variables and both mechanisms and outcomes. These reflected a range of

individual-level contextual considerations that were determined to be relevant for families,

including age, service use, number and type of mental health and/or addiction concerns, and

reasons for seeking navigation. Qualitative results further expanded on how and why contexts

impact families, both at the individual- and system-level. Age and type of mental health and/or

addiction concern, for example, may have created contexts in which, despite the efforts of the

family and service providers, the youth in question refuses to engage; and at the systemic level,

flexible interpretations of legislation may have created contexts in which families and service

providers were less able to effectively collaborate on the youth’s behalf. A larger and more

significant context for families in this sample was the systemic lack of high quality services for

youth MHA concerns, particularly if youth had concurrent concerns or was transitional-aged.

Throughout this study, the importance of understanding and accounting for context was

consistently reaffirmed.

However, qualitative results did suggest that some refinement of the program theory and

conceptual framework, particularly with regard to the range of mechanisms underlying family

navigation and how they interact, was necessary. Chapter 2 discussed the underpinnings of

realist evaluation theory, which presumes that social programs like the FNP generate outcomes

by providing resources (e.g. information, skills, or support) and/or influencing their participants’

reasoning (e.g. values, beliefs, or attitudes), which goes on to influence decision-making and

subsequent outcomes (Pawson & Tilley, 1997). More recent work in realist evaluations has

reiterated the importance of specifically identifying mechanisms as related to either resources or

Page 140: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

127

reasoning, which allows for clearer specification of how resources and reasoning are influenced

by context, and how this is associated with the given outcomes (Dalkin, et al., 2015).

The qualitative results of this study first indicated that beyond accessible, continuous, and

family-inclusive services, several other “resources” were at play. The first was specific

knowledge, insight and expertise in youth mental health and addictions. Families benefitted

significantly from having access to a Navigator who had a strong understanding of the conditions

themselves, the range of available service options, and the nuances of the service systems. In the

original conceptual framework (Table 1, Appendix B), “accessible expertise” was listed as the

first defining component of family navigation. The underlying mechanism was proposed to be

the accessibility of expertise, but the results of this study suggested that in such a complex

service system, expertise itself is a specifically important resource for families. Clients in the

qualitative sample spoke about how their Navigator’s expertise led them to make the most

appropriate recommendations and referrals for that client’s particular needs. Qualitative data

maintained accessibility was also an important mechanism for families, but that the concept had

potentially been overextended in the original framework; clients tended to speak about

accessibility specifically in regard to their ability to tangibly connect with the program and

receive attention as needed, quickly and efficiently.

The second additional resource to which the qualitative results pointed was family support.

Family support was a theme assumed to fall under family involvement based on the definition

employed, but which instead emerged from the data, suggesting it was a unique resource that

may be more applicable to the framework than family involvement. In fact, few clients

commented specifically on their involvement in the process. Instead, families spoke more

generally about the program’s family-centredness, and specifically about support. It appears that

families may have sought, received and benefitted from a family perspective and family support,

but may not have placed as much emphasis on being involved or did not necessarily want to be

actively involved in the process. The results suggest that the broader mechanism may thus be a

family-centred perspective, with one of the specific resources being the provision of family

support.

Qualitative data also suggested that in response to these mechanistic resources, several specific

reasoning processes occurred that were then linked to client outcomes. Clients who received

Page 141: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

128

expertise and/or support that they perceived as accessible, continuous, and/or family-centred

(according to their needs) felt significantly relieved, reassured, and/or more hopeful for the

future as a direct result. This increased sense of relief, reassurance and hopefulness represents a

change in clients’ reasoning that subsequently enabled families to better manage their daily lives

and the ongoing needs of their youth and family. The role of reasoning in elucidating how these

mechanistic resources generate changes in outcomes is compelling and important for

understanding how best to help families with youth who have mental health and/or addiction

concerns.

Altogether, the results of this study suggest several considerations for the original program

theory and conceptual framework. The refined program thus states that the Family Navigation

Project is situated amid a variety of influential individual- and systemic-level contexts, but that

with appropriate inputs, Navigators are enabled to provide accessible, continuous, family-centred

expertise and support to families of youth with mental health and/or addiction concerns, which

engenders a sense of relief, reassurance, and/or hopefulness that in turn, has a significant positive

impact on their level of family empowerment, FQOL, and service satisfaction.

Page 142: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

129

Chapter 9 Conclusions

Limitations, mitigations and contributions

9.1 Study limitations and mitigations

This study employed a mixed methods design within a RE framework in order to better

understand the impacts of the FNP on families’ level of empowerment, quality of life, and

service satisfaction as a mutable, complex product of biopsychosocial contexts and mechanisms.

However, there were several significant design limitations in this project. First and foremost, this

evaluation relied solely on families; it did not collect data from the youth themselves. This

deliberate decision was made for several reasons: 1) in the context of youth mental health and/or

addictions, families are understood to be the primary facilitator when interacting with the service

system; 2) youth mental health and/or addiction concerns are understood by the program to be a

family disease; and 3) youth mental health and/or addiction concerns have incredibly unique and

complex pathways, and are particularly difficult to treat, meaning significant causal associations

and effects are difficult to detect. Related to the reliance on family-focused outcome measures

was the assumption that one family member’s response adequately reflects the family as a whole.

Similarly, it is important to note that in the current study, reported MHA concerns were those

that are parent-perceived, which is conceptually distinct from a formally diagnosed mental

disorder. Future research should strive to collect additional family members’ perspectives, youth

perspectives, and navigator perspectives as well.

The current study evaluated the role of context at the individual/family level, and at the systemic

level. However, organizational-level contextual factors should be considered in future research,

as there are undoubtedly unique program qualities to which desired outcomes may be attributed.

For example, the FNP is privately funded, which allows it to operate independently and outside

restrictions placed on publicly funded programs. For example, the FNP is not required to operate

within a particular catchment area, and can thus accept clients from across the GTA. Similarly,

because they are not publicly funded, they are not mandated to refer only to public services. This

allows them to refer families to the full range of available resources. This is particularly

Page 143: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

130

important to families who are seeking care for MHA concerns for which there are few accessible

public options available (eating or personality disorders, for example).

Another main limitation was the lack of comparator group. Because this program was both new

and unique, a cross-sectional design relying on self-report data, without a comparison or control

group was necessary. However, employing an RE framework mitigated these drawbacks to some

extent by prioritizing patient-reported outcomes and a robust understanding of context, which

was believed to be more appropriate and pragmatic, given the family-focused goals of the study.

History and maturation threats to validity still exist though, as “navigation” has become a

buzzword and policy changes that affect empowerment, quality of life, and satisfaction scores

across the system may have arisen since the study was first conceptualized. Mental health and

addiction conditions are also evolving and recurrent in nature, especially among youth;

perceptions and outcomes may have been influenced by the natural progression of the illness(es)

and accumulation of missed opportunities. Again, RE attempted to mitigate this with a thorough

description of context (Pawson & Tilley, 1997). Nonetheless, it would be interesting to pursue a

longitudinal design in the future.

The results of the study indicated that the overall sample of youth was slightly older than

expected; the majority of the sample was over the age of 19. As such, it is likely that the

“child/youth” language in some outcome measures, such as the FES, may not have been as

appropriate as anticipated. Some clients specifically noted in qualitative fields that specific

questions, such as those related to school work or extracurricular activities, were inappropriate.

There are, however, limitations with the qualitative data as well, particularly in terms of the

manner in which it was collected. Open-ended prompts following each of the quantitative

measures were originally intended to gather supplemental qualitative data that would help to

contextualize and further understand respondents’ scores. However, these prompts unexpectedly

yielded spontaneous, highly extensive, and rich descriptions of families’ experiences. While the

originally proposed study design included semi-structured in-person interviews as a follow up to

survey responses, the selected theoretical framework - Realist Evaluation - strongly encourages

adaptability and responsiveness throughout the course of an evaluation. As such, following a

discussion with the program team and the PI’s supervisory committee with regard to 1) the

surprising richness and quality of the qualitative data resulting from the open prompts; and 2) the

Page 144: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

131

ethical considerations related to the burden of asking families to attend in-person interviews after

so many had voluntarily provided rich accounts that touched on many of the proposed interview

topics, a decision was made by the PI to proceed with a descriptive analysis of the qualitative

data resulting from the open prompts.

RE as an evaluative framework also has its limitations. Because of the methodological

flexibility, there is a lack of clarity and explicit direction in the field around how to generate C-

M-O configurations from the data (Dalkin, et al., 2015). However, the current study proposed

and expanded upon a method that has previously been clearly articulated in the literature (Byng,

et al., 2005); and it is the hope of the PI that the current study will serve as another example of

how C-M-O configurations can be constructed from both quantitative and qualitative data.

This this study (and RE studies in general) resulted in data that is highly contextualized and as

such, is not intended to be generalizable to other settings. This is not to say it will not still be

useful. To enhance external validity, the Theory of Proximal Similarity was used to ensure the

context and sample itself was extensively described and analyzed in relation to both mechanisms

and outcomes such that future researchers have the information necessary to adequately interpret

the relevance and implications of the findings in light of their specific contexts (Campbell &

Stanley, 1963). This will be particularly relevant for other jurisdictions looking to implement

similar service models.

External validity was deprioritized in favour of a high degree of internal validity in this study.

Extensive efforts were undertaken to ensure that the underlying program theory, study

objectives, research question and methods were stakeholder-driven and appropriately validated.

This approach, along with the use of mixed methods to mitigate the limitations of each individual

method and enhance robustness of the data, although time-intensive, lent heavily to the internal

validity and utility of the study. Other favourable aspects included the participatory nature of the

population and the quickly growing roster of registered families, which helped to minimize

concerns about adequate survey response rates and sample sizes. It is noted, though, that the

overall response rate in this study was quite low. Although inferential models had sufficient

sample sizes to accommodate the proposed predictors for each outcome, a larger sample would

have provided increased opportunity to detect individual differences in score; and perhaps the

three individual mechanisms could have been tested separately.

Page 145: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

132

Another limitation is the fact that this study employed a convenience sample that was likely

biased against families with more challenging contexts that would deter participation; and the

contexts that deter participation could similarly be the contexts in which less positive

experiences and outcomes may have been observed. Similarly, it is possible that the sample

opting to provide qualitative data was biased toward families whose contexts, experiences and/or

outcomes were more complex, compelling them to volunteer additional qualitative information.

In addition, as the program has grown and extended its reach in throughout the GTA, the

clientele is expected to have evolved as well. This expectation was confirmed by program team,

who indicated that while most descriptions of the study sample aptly described their current

client roster (such as the types of reported MHA concerns, proportion of concurrent cases and

average length of stay), other characteristics of the study sample (namely the high proportion of

Caucasian clients and those with high household incomes) were no longer true. As such, the

results of this study are not considered generalizable outside the study sample.

Quantitative mechanism and outcome measures were employed so that inferential modelling of

the desired outcomes could be performed. However, there are limitations in using quantitative

measures to capture complex processes and outcomes like family empowerment and quality of

life as quantitative measures may not have sufficiently accounted for the wide range of possible

internal and external influences on an individual response. As a result, in modelling the

quantitative responses, identifiable relationships were generally quite modest in effect size. It is

likely the case that these concepts were better captured and elucidated qualitatively in present

study. Also, qualitative feedback could inform quantitative tools that may better capture family

experience. In this study, modelling results are believed to be sufficiently supplemented by the

qualitative data.

9.2 Contributions

The current study has important implications and makes several contributions to the field.

Internally, the results of this study offer an evaluation of the FNP’s efforts by highlighting

several service strengths, as well as opportunities for quality improvement. It also offers the FNP

conceptual and measurement frameworks populated with baseline data, which will be used to

inform and/or supplement future evaluations. These frameworks are worthwhile contributions to

the literature base as well; recall from the rationale for the current study (Section 3.3) that

Page 146: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

133

navigation is increasingly being employed across a range of health care settings, despite the lack

of literature around conceptualization and evaluation. This study has yielded both a conceptual

framework for family navigation and a mixed methods measurement framework for evaluation,

as well as a series of testable program theories to guide future research. Lastly, these frameworks

offer a model for family navigation that can help with implementation in other jurisdictions,

which was a stated program goal. The proposed navigation continuum of care may be

particularly useful for similar or emerging navigation initiatives as a means of locating their

particular initiative within the wider spectrum of navigation service models.

This study also contributes to the evaluation literature by applying a relatively novel framework

to a novel context. In doing so, it generates another example of how and why Realist Evaluation

can be appropriate for evaluating complex health interventions. As mentioned previously, this

study also contributes to the evaluation literature a clearly articulated method for constructing C-

M-O configurations.

Most importantly, this study thoroughly explains what family navigation is and demonstrates its

role in the system. The results of this study lend support to the value of the FNP’s service model

as viable way in which we can practically help families navigate the system in order to better

access the right care, in the right place, at the right time. It also provides an in-depth description

of who, in the GTA, is currently using navigation services and for what reasons. In doing so, it

highlights some of the major barriers families are facing in accessing appropriate mental health

and/or addiction care for their youth. This is important information for future service planning.

Further, the study contributes to a literature base in which academic studies on both navigation

and RE are very much limited. The hope is that readers of this study will develop a detailed and

nuanced understanding of the local youth mental health and/or addiction services systems; what

family navigation is; who is using it and for what reasons; and how and why it helps families.

Page 147: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

134

References

Al-Abri, R., & Al-Balushi, A. (2014). Patient satisfaction survey as a tool toward quality

improvement. Oman Medical Journal, 29(1), 3-7.

Anderson, J.E., & Larke, S.C. (2009). Navigating the mental health and addictions maze: A

community-based pilot project of a new role in primary mental health care. Mental

Health in Family Medicine, 6, 15-19.

Armstrong, R.A. (2014). When to use the Bonferroni correction. Journal of the College of

Optometrists, 34(5), 502-508.

Bakas, T. (2014). Bakas caregiving outcomes scale. Encyclopedia of Quality of Life and Well-

Being Research, pp. 319-321. Netherlands: Springer.

Barr, J.K., et al. (2006). Using public reports of patient satisfaction for hospital quality

improvement. Health Services Research, 41(3 Pt 1), 663-683.

Battaglino, L. (1987). Family empowerment through self-help groups. New Directions for

Mental Health Services, 34, 43-51.

Bhaskar, R. (1979). The possibility of naturalism: A philosophical critique of the contemporary

human science. Atlantic Highlands, NJ: Humanities Press.

Boyd, R.N. (1989). What realism implies and what it does not. Dialectica, 43, 5-29.

Bryson, J.M. (2011). Strategic planning for public and nonprofit organizations: A guide to

strengthening and sustaining organizational achievement (Volume 1). New Jersey: John

Wiley & Sons.

Byng, R., et al. (2005). Using realistic evaluation to evaluate a practice-level intervention to

improve primary healthcare for patients with long-term illness. Evaluation, 11 69-93.

Cabin, R.J., & Mitchel, R.J. (2000). To Bonferroni or not to Bonferroni: When and how are the

questions. Bulletin of the Ecological Society of America, 81(3), 246-248.

Campbell, C., et al. (2010). Implementing and measuring the impact of patient navigation at a

comprehensive community cancer center. Oncology Nursing Forum, 37(1), 61-68.

Campbell, D.T. & Stanley, J.C. (1963). Experimental and quasi-experimental designs for

research. Chicago, IL: Rand McNally.

Canadian Institute for Health Information. (2013). Hospital mental health services for

concurrent mental illness and substance use disorders in Canada. Retrieved from

https://www.cihi.ca/en/hospital-mental-health-services-for-concurrent-mental-illness-

and-substance-use-disorders-in-canada

Canadian Institute for Health Information. (2017). Child and youth mental health in Canada –

Infographic. Retrieved from https://www.cihi.ca/en/child-and-youth-mental-health-in-

canada-infographic

Canadian Mental Health Association. (2006). Caring together: Families as partners in the

mental health and addiction system. Retrieved from

https://ontario.cmha.ca/documents/caring-together-families-as-partners-in-the-mental-

health-and-addiction-system/

Page 148: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

135

Canadian Mental Health Association. (2013). Concurrent disorder services in Ontario: An

environmental scan. Retrieved from https://ontario.cmha.ca/documents/concurrent-

disorder-services-in-ontario-an-environmental-scan/

Center for Behavioral Health Statistics and Quality. (2015). Behavioral health trends in the

United States: Results from the 2014 National Survey on Drug Use and Health (HHS

Publication No. SMA 15-4927, NSDUH Series H-50). Retrieved from

http://samhsa.gov/data

Children’s Mental Health Ontario. (2016a). Breaking point – a system stretched beyond its

limits: A report on community-based children’s mental health centres. Retrieved from

http://cmho.org/education-resources/cmho-s-latest-work/item/511-breaking-point-a-

system-stretched-beyond-its-limits

Children’s Mental Health Ontario. (2016b). Residential treatment: Working towards a new

system framework for children and youth with severe mental health needs. Retrieved

from https://www.kidsmentalhealth.ca/fr/education-resources/cmho-s-latest-work

Chovil, N. (2009). Engaging families in child and youth mental health: A review of best,

emerging and promising practices. Vancouver, BC: F.O.R.C.E. Society of Kids’ Mental

Health. Retrieved from

http://www.excellenceforchildandyouth.ca/sites/default/files/resource/EIS_Family_Engag

ement_EN.pdf

Conference Board of Canada. (2014). Final report: An external evaluation of the Family Health

Team (FHT) initiative. Retrieved from http://www.conferenceboard.ca/e-

library/abstract.aspx?did=6711

Corporate Research Associates Inc. (2004). Cancer patient navigation evaluation: Final report.

Nova Scotia, Canada: Cancer Care Nova Scotia. Retrieved from

http://www.cancercare.ns.ca/site-

cc/media/cancercare/patientnavigationevaluationfindings.pdf

Curtin, F., & Schulz, P. (1998). Multiple correlations and Bonferroni’s correction. Biological

Psychiatry, 44, 774-777.

Curtis, W.J., & Singh, N.N. (1996). Family involvement and empowerment in mental health

service provision for children with emotional and behavioural disorders. Journal of Child

and Family Studies, 5(4), 503-517.

Dalkin, S.M., Greenhalgh, J., Jones, D., Cunningham, B., & Lhussier, M. (2015). What’s in a

mechanism? Development of a key concept in realist evaluation. Implementation Science,

10, 49.

Dillman, D.A., et al. (2014). Internet, phone, mail and mixed-mode surveys: The Tailored Design

Method. 4th ed. Canada: Wiley.

Driscoll, D.L., Appiah-Yeboah, A., Salib, P., & Rupert, D.J. (2007). Merging qualitative and

quantitative data in mixed methods research: How to and why not. Ecological and

Environmental Anthropology (University of Georgia), 3(1), 19-28.

Dunst, C.J., Trivette, C.M., Davis, M., & Cornwell, J. (1988). Enabling and empowering families

of children with health impairments. Children’s Health Care, 17(2), 71-81.

Page 149: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

136

Field, A.P. (2009). Discovering statistics using SPSS: (and sex and drugs and rock ‘n’ roll).

Thousand Oaks, CA: SAGE Publications.

Fillion, L., et al. (2006). Implementing the role of patient navigator nurse at a university hospital

centre. Canadian Oncology Nursing Journal, 16(1), 11-17.

Fishman, K., & Levitt, A. (2014). Unpublished. Available upon request.

Freeman, H. (2011). As quoted in: Fayerman, P. (2011, February 25). The father of patient

navigation. The Vancouver Sun. Retrieved from

http://www.vancouversun.com/health/father+patient+navigation/4305453/story.html

Freeman, H.P., & Rodriguez, R.L. (2011). History and principles of patient navigation. Cancer,

117(suppl 15), 3527-2540.

Freund, P.D. (1993). Professional role(s) in the empowerment process: “Working with” mental

health consumers. Psychosocial Rehabilitation Journal, 16(3), 65-73.

Gill, T., & Renwick, R. (2007). Family quality of life and service delivery for families with

adults who have developmental disabilities. Journal on Developmental Disabilities,

13(3), 13-36.

Green, S.B. (1991). How many subjects does it take to do a regression analysis? Multivariate

Behavioral Research, 26, 499-510.

Haggerty, J.L., et al. (2003). Continuity of care: A multidisciplinary review. British Medical

Journal, 327, 1219-1221.

Hardin, J., & Hilbe, J. (2007). Generalized linear models and extensions. 2nd ed. College Station:

Stata Press.

Hoffman, L., Marquis, J., Poston, D., Summers, J.A., & Turnbull, A. (2006). Assessing family

outcomes: Psychometric evaluation of the beach centre family quality of life scale.

Journal of Marriage and Family, 68(4), 1069-1083.

Hook, A., Ware., L., Siler, B., & Packard, A. (2012). Breast cancer navigation and patient

satisfaction: Exploring a community-based patient navigation model in a rural setting.

Oncology Nursing Forum, 39(4), 379.

IBM Corporation. (2013). IBM Statistical Package for the Social Sciences, Version 11.0 for Mac.

Armonk, NJ: IBM Corp.

Institute for Clinical Evaluative Sciences. (2017). The mental health of children and youth in

Ontario: 2017 scorecard. Retrieved from https://www.ices.on.ca/Publications/Atlases-

and-Reports/2017/MHASEF

Isaacs, B.J. et al. (2007). Development of a family quality of life survey. Journal of Policy and

Practice in Intellectual Disabilities, 4, 178.

Kessler, R.C., et al. (2005). Lifetime prevalence and age-of-onset distributions of DSM-IV

disorders in the National Comorbidity Survey Replication. Archives of General

Psychiatry, 62(6), 593-602.

Kessler, R.C., et al. (2012). Lifetime comorbidity of DSM-IV disorders in the NCS-R

Adolescent Supplement (NCS-A). Psychological Medicine, 42(9), 1997-2010.

Page 150: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

137

Klein, D.M., & White, J.M. (1996). Family theories: An introduction. Thousand Oaks, CA: Safe

Publications, Inc.

Kopp, J. (1989). Self-observation: An empowerment strategy in assessment. Social Casework,

70, 276-284.

Koren, P.D., DeChillo, N., & Friesen, B.J. (1992). Measuring empowerment in families whose

children have emotional disabilities: A brief questionnaire. Rehabilitation Psychology,

37(4), 305-321.

Koren, P.E., et al. (1997). Service coordination in children’s mental health: An empirical study

from the caregiver’s perspective. Journal of Emotional and Behavioural Disorders, 5,

162-172.

Lazear, K., Worthington, J., & Detres, M. (2004). Findings compendium: Issue brief 5,

Helpfulness of formal services, family organizations and informal supports. Tampa,

Florida: University of South Florida, Louis de la Parte Mental Health Institute, Research

Training Centre for Children’s Mental Health.

Leplin, J. (1981). Truth and scientific progress. Studies in History and Philosophy of Science, 12,

269-292

Lindsey, J.K., & Jones, B. (1998). Choosing among generalized linear models applied to medical

data. Statistics in Medicine, 17, 59-68.

McAllister, M., Dunn, G., Payne, K., Davies, L., & Todd, C. (2012). Patient empowerment: The

need to consider it as a measureable patient-reported outcome for chronic conditions.

BMC Health Services Research, 12, 157.

McCammon, S.L., Spencer, S., & Friesen, B.J. (2001). Promoting family empowerment through

multiple roles. Journal of Family Social Work, 5, 1-24.

McCullagh, P., & Nelder, J.A. (1989). Generalized linear models. 2nd ed. London: Chapman and

Hall.

McDonald, K.M., et al. (2007). Volume 7: Care coordination. In Shojania, K.G., et al. (eds).

Closing the quality gap: A critical analysis of quality improvement strategies (technical

review 9). Rockville, MD: Agency for Healthcare Research and Quality. Retrieved from

https://www.ncbi.nlm.nih.gov/books/NBK44015/

McPhee, J., Syed, T., Nunes, M., & the Mobilizing Minds Research Group. (2012). An

environmental scan of knowledge translation, mental health literacy, and policy/system

change initiatives aimed at improving the mental health of youth and young adults in

Canada. Retrieved from https://mindyourmind.ca/sites/default/files/Full%20Report%20-

%20Environmental%20Scan.pdf

Mental Health Commission of Canada. (2012). Changing directions, changing lives: The mental

health strategy for Canada. Retrieved from http://strategy.mentalhealthcommission.ca

Mental Health Commission of Canada. (2015a). Informing the future: Mental health indicators

for Canada. Retrieved from

https://www.mentalhealthcommission.ca/English/document/68796/informing-future-

mental-health-indicators-canada

Page 151: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

138

Mental Health Commission of Canada. (2015b). The mental health strategy for Canada: A youth

perspective. Retrieved from

https://www.mentalhealthcommission.ca/English/document/72171/mental-health-

strategy-canada-youth-perspective

Mental Health Commission of Canada. (2017). Strengthening the case for investing in Canada’s

mental health system: Economic considerations. Retrieved from

https://www.mentalhealthcommission.ca/English/case-for-investing

Messick, S. (1990). Validity of test interpretation and use. Research Report 90-11. Education

Testing Service.

Miller, R.W. (1987). Fact and method: Explanation, confirmation and reality in the natural and

social sciences. Princeton: Princeton University Press.

Ministry of Health and Long Term Care. (2005). FHT guide to collaborative team practice.

Retrieved from

https://scele.ui.ac.id/berkas_kolaborasi/konten/MKK_2014genap/family_team.pdf

Ministry of Health and Long Term Care. (2009). Every door is the right door: Towards a 10-

year mental health and addictions strategy. A discussion paper. Retrieved from

http://ontario.cmha.ca/wp-content/uploads/2016/08/Every-Door-the-Right-Door-July09-

MH-discussion-paper.pdf

Ministry of Health and Long Term Care. (2011). Open minds, healthy minds: Ontario’s

comprehensive mental health and addictions strategy. Retrieved from

http://health.gov.on.ca/en/common/ministry/publications/reports/mental_health2011/men

talhealth.aspx

Mpinga, E.K., & Chastonay, P. (2011). Satisfaction of patients: A right to health indicator?

Health Policy, 100(2-3), 144-150.

Nguyen, C.T., Fournier, L., Bergeron, L., Roberge, P., & Barrette, G. (2005). Correlates of

depressive and anxiety disorders among young Canadians. Canadian Journal of

Psychiatry, 50(10), 620-628.

Nunnally, J.C. (1978). Psychometric theory. 2nd ed. New York: McGraw Hill.

Ontario Hospital Association. (2016). A practical guide to mental health and the law in Ontario.

Retrieved from

https://www.oha.com/Legislative%20and%20Legal%20Issues%20Documents1/A%20Pr

actical%20Guide%20to%20Mental%20Health%20and%20the%20Law%20in%20Ontari

o%20%282012%29%20%28PUBLICATIONS%29.pdf

Ontario Human Rights Commission. (2015). By the numbers: A statistical profile of people with

mental health and addiction disabilities in Ontario. Retrieved from:

http://www.ohrc.on.ca/en/numbers-statistical-profile-people-mental-health-and-

addiction-disabilities-ontario

Park, J., et al. (2003). Toward assessing family outcomes of service delivery: Validation of a

family quality of life survey. Exceptional Children, 68, 151-170.

Pautler, K. (2005). Annotated bibliography of collaborative mental health care. Mississauga,

Ontario: Canadian Collaborative Mental Health Initiative.

Page 152: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

139

Pawson, R., & Tilley, N. (1997). Realist Evaluation. London: Sage.

Pearson, C., Janz, T., & Ali, J. (2013). Mental and substance use disorders in Canada. Health at

a Glance. (Catalogue no. 82-624-X). Ottawa, ON: Statistics Canada.

Pedersen, A., & Hack, T. (2010). Pilots of oncology health care: A concept analysis of the

patient navigator role. Oncology Nursing Forum, 37(1), 55-60.

Penchansky, R., & Thomas, W.J. (1981). The concept of access: Definition and relationship to

consumer satisfaction. Medical Care, 19(2), 127-140.

Putnam, H. (1982). Three kinds of scientific realism. Philosophical Quarterly, 32, 195-200.

QSR International Pty Ltd. (2015). N. Vivo qualitative data analysis software v.10 for Mac.

Renwick, R., Brown, I., & Raphael, D. (1998). The family quality of life project: Final report.

Report to the Ontario Ministry of Community and Social Services. Toronto, ON: Centre

for Health Promotion, University of Toronto. Retrieved from

http://sites.utoronto.ca/qol/projects/pwdd.htm

Rhemtulla, M., Brosseau-Liard, P.E., & Savalei, V. (2012). When can categorical variables be

treated as continuous? A comparison of robust continuous and categorical SEM

estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354-

373.

Richardson, G.P., & Anderson, D.F. (1995). Teamwork in group model building. System

Dynamics Review, 11(2), 113-137.

Richter, P. (2017). Access to OCD treatment is limited. And that’s a problem. The Huffinton

Post. Retrieved June 14, 2017 from http://www.huffingtonpost.ca

Roberts, A., & Schmidt, N. (2012). The Family Navigation Project business case. Toronto, ON:

Author.

Robinson-White, S., Conroy, B., Slavish, K.H., & Rosenzweig, M. (2010). Patient navigation in

breast cancer: A systematic review. Cancer Nursing, 33(2), 127-140.

Rogers, P.J. (2008). Using programme theory for complicated and complex programmes.

Evaluation, 14(1), 29-48.

Samuel, P.S., Rillotta, F., & Brown, I. (2012). The development of family quality of life

concepts and measures. Journal of Intellectual Disability Research, 56(2), 1-16.

Samuel, P.S., Rillotta, F., & Brown, I. (2012). The development of family quality of life

concepts and measures. Journal of Intellectual Disability Research, 56(2), 1-16.

Sandelowski, M. (2000). Whatever happened to qualitative description? Research in Nursing &

Health, 23, 334-340.

Seek, A., & Hogle, W.P. (2007). Modeling a better way: Navigating the healthcare system for

patients with lung cancecr. Clinical Journal of Oncology Nursing, 11(1), 81-85.

Singh, N.N. (1995). In search of unity: Some thoughts on family-professional relationships in

service delivery systems. Journal of Child and Family Studies, 4, 3-18.

Sirotich, F., & Durbin, A. (2014). Identifying the needs of complex health populations receiving

community mental health and addictions services: An analyses of Ontario Common

Page 153: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

140

Assessment of Need (OCAN) data for case management and supportive housing

programs. Final report. Toronto, ON: Canadian Mental Health Association.

Smith, J.P., & Smith, G.C. (2010). Long-term economic costs of psychological problems during

childhood. Social Science & Medicine, 71(1), 110-115.

Staples, L.J. (1990). Powerful ideas about empowerment. Administration in Social Work, 14,

p.30.

Statistics Canada. (2006). The human face of mental health and mental illness in Canada.

Retrieved from http://www.phac-aspc.gc.ca/publicat/human-humain06/index-eng.php

Studenmund, A.H. (2010). Using econometrics: A practical guide. New Jersey: Harlow Pearson

Education.

Tabachnick, B.L., & Fidell, L.S. (2007). Using multivariate statistics. 5th ed. Boston, MA:

Pearson Education, Inc.

Tannenbaum, L.G. (2001). Parent/professional perceptions of collaboration when viewed in the

context of Virginia’s Comprehensive Services Act system of care. Dissertation.

Blacksburg, VA: Virginia Polytechnic Institute and State University. Retrieved from

http://theses.lib.vt.edu/theses/available/etd-12192001-

122745/unrestricted/TannenbaumRevised.pdf

Turnbull, A.P., et al. (2000). Enhancing quality of life for families of children and youth with

disabilities in the United States. In Turnbull, A., Brown, I., & Turnbull, R.H. (eds).

Families and People with Mental Retardation and Quality of Life: International

Perspectives. Washington, DC: American Association on Mental Retardation.

VanVoorhis, C.R.W., & Morgan, B.L. (2007). Understanding power and rules of thumb for

determining sample sizes. Tutorials in Quantitative Methods for Psychology, 3(2), 43-50.

Vennix, J.A.M. (1996). Group model building: Facilitating team learning using system

dynamics. Chichester: John Wiley & Sons.

Ware, J.E., Synder, M.K., Wright, W.R., & Davies, A.R. (1983). Defining and measuring patient

satisfaction with medical care. Evaluation and Program Planning, 6(3-4), 247-263.

Wood, G.M. (2004). Health care reform tracking project (HCRTP): Promising approaches for

behavioral health services to children and adolescents and their families in managed care

systems. Family involvement in managed care systems (FMHI Publication #211 – p.6).

Tampa, FL: Louis de la Parte Florida Mental Health Institute, University of South

Florida.

Yong, A.G., & Pearce, S. (2013). A beginner’s guide to factor analysis. Tutorials in Quantitative

Methods for Psychology, 9(2), 79-94.

Page 154: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

141

Page 155: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

142

Appendix A Themes from the Family Navigation Project’s Focus Groups1

1. Access to accurate information: Parents want access to good, objective information

about the nature of the problem and about treatment options and various providers.

a. They want realistic information about their child’s difficulties and up-to-date

information about the range of treatment options and possible outcomes.

b. Parents also want help in weighing and evaluating the information and

recommendations that they are receiving from different sources.

c. Parents want to know about preventative resources earlier on and would like

guidance about when and how strongly to advocate for treatment. As examples they

posed these questions: “Do you wait until your child is in crisis? At what point do

you accept that there is a problem and that something must be done? How active do

you get in pushing the alternative of treatment?”

2. Guidance and support: Parents would like to have a “live person” involved in the person

of a mentor or guide who will help them to answer critical questions about the process.

a. They want the opportunity to review the information they are receiving with an

experienced and objective professional who can help them to assess the information

and figure out what would be a good fit for their child and themselves.

b. They would like to have someone on board whose job it is to listen to their concerns

and provide direction and support.

c. Parents feel that it is not enough to put a hot line in place – they need to be able to

talk to a responsive, interactive and caring person from the time they make the initial

call.

d. As a possible prototype of the kind of navigator they would like to see, parents

identified a professional group in the US known as Therapeutic and Educational

Placement Specialists (also known as Educational Consultants) who consult to

1 Sourced from the Family Navigation Project Business Case (Roberts, A. & Schmidt, N., 2012

Page 156: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

143

parents about treatment possibilities and options within therapeutic programs such as

therapeutic boarding schools, wilderness programs and residential treatment centres.

3. Consistent, ongoing involvement: Parents describe feeling as if they are travelling on a

rudderless ship without a compass and with little experience or knowledge of what to

expect to guide their way. They wonder, “Where do you go? What is out there? To whom

do you turn?” They want a guide who is “hands on” and will “get in the boat with them” to

help them navigate their way through the process. They want someone who will stay

involved.

a. Parents want solid information about the barriers they are likely to meet along the

way and how they can go about circumventing them.

b. They want help in changing course if they or the youth discovers that the program or

therapist is not the right one. Parents want to know what to do next if the situation is

not improving.

4. Meaningful role and participation: Most importantly, most parents want to be actively

engaged in their child’s care. However, they struggle to know how to be involved as a

parent – e.g., whether to be more active or to back off and just be there for the youth.

They generally know that there is learning and healing that needs to take place on their

own part, but may not know where to start or who would help them.

a. Parents want someone to talk to about what they are learning as they go through the

process.

b. Parents want to be meaningfully involved in their child’s treatment program. They

want to participate in providing information and feedback to the evaluators and they

are deeply interested in receiving information about whether the child is attending

and progress is being made.

c. Most parents want support for themselves in order to cope with the ongoing

situation, and many want help in learning more about their own role in the situation

and potentially in the solution.

5. Tailoring and matching of approach: Parents and the people with the mental health

concerns want information about the nuance of a particular treatment approach or provider

Page 157: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

144

and an understanding of whether or not a particular therapist and/or approach is likely to

be a good match for them or their child. Often they are looking for something different or

something more after trying things on their own or seeking help through the family doctor,

the school or a clinic. In spite of everyone’s best efforts, things are not improving. In

particular, they would like help in answering the following questions:

▪ How do we know whether a particular therapist, treatment provider or program is

right for our child?

▪ How will we as parents be involved and what kind of information about progress can

we expect to receive? Will we be informed regularly?

▪ Who will be our child’s advocate as we go through this? Who will talk to us?

▪ How will we know if things are not working out? How will we go about changing

course if a particular therapeutic approach or program is not working out?

6. Attention and support at critical junctures: Parents especially look for guidance and

support at critical junctures such as an inpatient admission or when, after all kinds of

efforts by the family and professionals alike, it is apparent that out-of-home residential

treatment needs to be contemplated.

a. With any kind of group therapy program whether in a residential or hospital setting,

parents want to know what the peer influences will be. They want to know who will

be in the program with their child and how to be sure that the peer culture is a

positive one and that the child is not going to get worse.

b. Parents also want to be sure that the program is not going to be punitive (e.g.,

wanting to be sure “that we are not sending our child to a prison”).

c. They want to know how visitation times/periods work, whether they will be involved

in family therapy sessions, and – in the case of a distant residential treatment

program – how they will be meaningfully involved (e.g., through telephone or Skype

family sessions and/or therapeutic letter-writing and/or on-site visits).

d. Especially in the case of a distant program, parents want assurance that they will be

informed if things are or are not working out and receive help in changing course, if

necessary.

Page 158: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

145

e. Parents whose youth are in inpatient or residential programs would like to receive

help with aftercare or transition planning for when the youth is to leave the program

and/or go home.

7. Peer support: Parents want access to other parents who have been through similar

experiences and can envision groups of parents who have been through the process acting

as resources to others. In particular, contact with other parents can help people get past

stigma, denial and embarrassment so that these issues do not become barriers in their lives.

8. Accessibility: Parents want access to help outside of normal business hours; “crisis don’t

occur between 9-5pm” was a common sentiment. This means after-hours clinics or crisis

lines, mobile consultations, ability to communicate by any means available (Skype, email).

They also want a service whose offerings are not limited solely to those in the

neighbourhood. And lastly, they want a service facilitates equal access for everyone, not

just those with the financial resources for private treatment.

9. Options: Parents are looking for information about the full range of possible treatment

options including local private practitioners and private treatment programs and US and

international options.

a. Parents would like information about treatment programs that offer alternatives to

the current orthodoxies of the public system (e.g., they are interested in learning

about programs with abstinence-based rather than harm reduction approaches to

substance abuse treatment and about relationship-based residential treatment rather

than strictly behavioural approaches).

b. Parents would like to know about private and fee-for-service options.

c. They would like help in connecting to US Therapeutic and Educational Placement

Specialists when needed.

d. They would like to have information regarding costs of private programs within

Canada and the U.S., about applying for reimbursement from OHIP for US

programs, and about consent issues locally and in relation to US residential facilities.

10. Use of evidence and data in decision-making: Parents would like to see a navigational

service that makes intelligent use of data by tracking outcomes over time in order to create

Page 159: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

146

a database of “collective wisdom”. In addition, they would like to see the development of

reliable, on-line resources for parents.

11. Acknowledgement and hope: Most importantly, parents want a place to go where they can

get acknowledgement that something is wrong and hope that there is a way forward. Part of this

will come from contact with other parents who have been through similar situations and have

found some answers and solutions. Part will come from learning that there is a range of options

and a critical pathway to help them move forward.

Page 160: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

147

Appendix B Logic Model and Conceptual Framework

Figure 1. Logic Model for the Family Navigation Project

Page 161: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

148

Table 1. Conceptual framework for family navigation

Sphere of

Influence Components Definition

Theorized

Mechanism

Family-

level

(service-

client

interaction)

Accessible expertise Services are organized to respond to families' needs; phone and email-based, extended and

flexible hours, meeting space; medical consultation and supervision available; Navigator

experience and expertise

Accessibility

Family education Navigators provide families with information and resources aimed at improving their

understanding of the problem itself, and the roles of the family, Navigators, and health care

system in the recovery process

Accessibility

Resource assessment

and information

sharing

Navigators provide appropriate and sufficient information on which resources are available

Accessibility

Resource matching Navigators identify the resources that best meet the family's identified needs Accessibility

Referrals Navigators directly connect families and facilitate relationships with identified resources or

services Accessibility

Connections to peer

support

Navigators provide families with connections to support networks of peers with lived

experience Accessibility

Family-based

perceived needs

assessment

Navigators discuss and determine the youth's and family's perceived needs and perceived

barriers to care Family

involvement

Family-based

collaborative care

planning

Navigators engage and work closely with families to develop care plans with clear steps and

supports that meet their perceived needs and overcome perceived carriers Family

involvement

Consistent family

engagement

Navigators consistently engage with families in all steps throughout the process Family

involvement

Information

dissemination and

Navigators ensure continuous and current information exchange across care settings so

families perceive transitions as seamless

Continuity of

care -

Page 162: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

149

exchange informational

Ongoing follow up

and response to

changing needs

Navigators communicate with families regularly to continuously monitor progress and adjust

course as necessary throughout the process

Continuity of

care -

management

Relationship and

trust-building

Navigators strive to develop a relationship and build rapport with families by providing care

and communicating in a professional, respectful and responsive manner

Continuity of

care -

relational

Service/Syst

em-level

Facilitate service-

system relationships

Navigators visit and build relationships with service providers in the system

Out of scope

Promote evaluation

and accountability

Navigators evaluate their own service and other resources in order to promote efficiency and

effectiveness in the system through evidence-based care

Promote education

and awareness for

youth MHA

The Family Navigation Project conducts research and shares findings at educational events

targeted at a wide range of stakeholders and settings

Advocate for youth

MHA and

navigation services

The Family Navigation Project advocates for its cause on a variety of platforms using

primary data

Competency training Navigators must be highly skilled and experienced mental health and/or addiction

professionals.

Page 163: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

150

Appendix C Measurement Summary

Table 1. Measurement summary: Context variables

Context measures

Variable Description Data type Collection

method Source

Client type Primary contact’s relation to youth Categorical

Chart

abstraction

Client

charts

Client status Clients are classified as active or inactive based on recency and frequency of

contact Dichotomous

Time since registration Number of weeks since initial registration Continuous

Age Age of youth for whom help is being sought Categorical

Gender Gender of youth for whom help is being sought Categorical

Mental health

Yes/no Parent-reported mental health concerns of youth for whom help is being

sought Dichotomous

Type(s) Parent-reported DSM-V category or dimension Categorical

Substance use

Yes/no Parent-reported substance use concerns of youth for whom help is being

sought Dichotomous

Type(s) Parent-reported DSM-V substance dependencies Categorical

Family/living situation Family structure and living arrangement of youth for whom help is being

sought Categorical

Education Education level of youth for whom help is being sought Categorical

Page 164: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

151

Legal involvement Presenting legal issues requiring involvement of external stakeholders Dichotomous

Catchment area Whether the youth for whom help is being sought resides inside or outside of

the SHSC catchment area Dichotomous

Prior service

use Yes/no Whether the client has previously received youth MHA services elsewhere Dichotomous

Previous ED

visit(s) Yes/no

Whether the youth of the client has had previous emergency department

visit(s) Dichotomous

Previous

inpatient stay(s) Yes/no Whether the youth of the client has had previous inpatient stay(s) Dichotomous

History of

bullying Yes/no Whether the youth has a reported history of being bullied Dichotomous

School

avoidance Yes/no Whether the youth has reported school avoidance Dichotomous

Reason for contact Reported reason(s) for contacting the program

Ethnicity of youth Self-reported ethnic background Categorical

Self-report

survey

Original

questions

Household income Self-reported household income Categorical

Youth functional status

since registration

Whether the youth’s general wellbeing has worsened, stayed the same, or

improved.

Categorical

Page 165: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

152

Table 2. Measurement summary: Mechanism variables

Mechanism measures

Variable Literature-based definition Original survey question Data

type

Accessibility

(Penchansky & Thomas,

1981)

Availability Adequacy of supply of existing services and

resources in relation to families’ needs.

To what extent do you feel you were able to

reach the Navigator whenever you needed?

Quan

tita

tive;

ord

inal

Accessibility

The relationship between the location of supply and

the location of families, taking account of factors

such as transportation resources and travel time,

distance, physical accessibility and cost.

To what extent do you feel it was convenient to

communicate or meet with the Navigator?

Accommodation

The relationships between how services and

resources are organized, such as appointment

systems, hours of operation, walk-in facilities,

telephone and web-based services; a family’s

relative ability to accommodate to these factors;

and their perception of appropriateness.

To what extent do you feel the Navigator

accommodated your schedule when making

arrangements with you?

Affordability

The relationship of cost of services (both the FNP

itself and the services to which it refers) to the

clients’ ability to pay. This includes “client

perception of worth relative to total

cost…knowledge of prices, total cost and possible

credit arrangements.”

To what extent do you feel the Navigator was

considerate of your financial resources?

Acceptability

The relationship between a family’s attitudes about

what the personal and practice characteristics

should be and the actual characteristics of the

service and its staff.

To what extent do you feel the Navigator met

your service expectations for family navigation?

Continuity of care

Informational

The use of personalized information on past events

and personal circumstances to make current care

appropriate for each family.

To what extent do you feel the Navigator

continuously communicated and coordinated

information with you and the service providers

to whom you were referred?

Page 166: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

153

(Haggerty et al., 2003)

Management

The use of a consistent and coherent approach in

managing care plans that respond to a family’s

evolving needs.

To what extent do you feel the Navigator

continuously adequately responded to changes

in your family’s situation and needs?

Relational The development of an ongoing therapeutic

relationship between a family and their navigator.

To what extent do you feel the Navigator was

continuously committed to understanding and

helping your family until you no longer feel you

require their services?

Family involvement

(Wood, 2004)

Respect for families as experts on their children;

enlisting them as partners in their child’s care;

supporting them in their caregiver role; and

involving them as partners in decision-making at all

levels.

To what extent do you feel the Navigator

consistently involved you and your family in all

stages of care planning and decision-making?

Page 167: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

154

Table 3. Measurement summary: Outcome variables

Outcome measures

Variable Definition Measure Data

type

Collection

method

Family

empowerment

(Koren,

DiChillo &

Friesen, 1992)

A family’s perception of itself

as having the knowledge, skills,

beliefs, attitudes and resources

required to successfully

navigate and negotiate the

service system and efficiently

utilize it to meet their needs.

Family Empowerment Scale (FES)

This bidimensional measure consists of 34 subjective statements scored on a 5-

point Likert-style scale which capture 1) the overall level of family empowerment;

and 2) and the way in which empowerment is expressed in daily life navigating the

service system to effectively and efficiently meet their needs.

The tool sums subscale item scores to produce a continuous total score for each

subscale.

Quan

tita

tive;

conti

nuous

Sel

f-re

port

surv

ey

Family quality

of life

(Hoffman,

Marquis,

Poston,

Summers &

Turnbull, 2006)

The degree to which families’

perceived needs are met, and to

which there are appropriate

opportunities to make active

choices that help them meet

their needs.

Modified Beach Center Family Quality of Life Scale (BCFQoLS)

Section A of this modified BCFQoLS consists of 20 items scored on a five-point

response scale ranging from “very dissatisfied (1)” to “very satisfied (5).” The

items span five conceptual domains of family quality of life: family interaction,

parenting, emotional well-being, physical/material well-being, and disability-related

support. Scores can be summed by domain and/or in total for a continuous measure.

A second section was been added to subjectively evaluate perceived change over

time since starting family navigation. This section consisted of 20 change items

which correspond to the 20 items in Section A and were rated on a scale of “not

true at all (1)” to “very true (3)” or “not applicable (4).” Scores can be summed by

domain and/or in total for a continuous measure.

Page 168: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

155

Service

satisfaction

(FNP, 2014)

The extent to which families

were satisfied with the

navigation services directly

received, as well as satisfaction

with the resources to which

families were referred by the

navigation team.

NAVSAT

This 25-item measure includes 15 items that ask families to assess their satisfaction

with the treatment recommendations received, their navigator’s knowledge and

fluency in the MHA system, respect for confidentiality, and nature/frequency of

contact; and 10 items that evaluate families’ satisfaction with the referred resource

in terms of type, delivery method, location and effectiveness.

The scale yields four outcome variables including likelihood of recommending the

service, navigator helpfulness, and overall service satisfaction. Items are rated on

five- or seven-point Likert-style scales. These first three outcome variables relate to

satisfaction with navigation services and were summed to a continuous total

outcome score. The fourth outcome variable is overall satisfaction with referred

service; this seven-point item was also treated as a continuous total score.

Page 169: A Realist Evaluation of Family Navigation in Youth Mental ......A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions Nadine Reid Doctor of Philosophy Institute

156