Building clinicians-researchers partnerships: Lessons from diverse natural settings and practice-oriented initiatives

This art icle was downloaded by: [ Society for Psychotherapy Research ]On: 04 Decem ber 2014, At : 11: 27Publisher: Rout ledgeI nform a Ltd Registered in England and Wales Registered Num ber: 1072954 Registered office: Mort im er House,37-41 Mort im er St reet , London W1T 3JH, UK

Psychotherapy ResearchPublicat ion det ails, including inst ruct ions for aut hors and subscript ion informat ion:ht t p: / / www. t andfonl ine.com/ loi/ t psr20

Building clinicians-researchers partnerships: Lessonsfrom diverse natural settings and practice-orientedinitiativesLouis G. Cast onguaya, Soo Jeong Youna, Henry Xiaoa, J. Christ opher Muranbc & Jacques P.

Barberb

a Depart ment of Psychology, Penn St at e Universit y, Universit y Park, PA, USAb Derner Inst it ut e for Advanced Psychological St udies, Adelphi Universit y, New York, NY, USAc Depart ment of Psychiat ry, Bet h Israel Medical Cent er, New York, NY, USAPubl ished onl ine: 01 Dec 2014.

To cite this article: Louis G. Cast onguay, Soo Jeong Youn, Henry Xiao, J. Christ opher Muran & Jacques P. Barber (2015)Building cl inicians-researchers part nerships: Lessons f rom diverse nat ural set t ings and pract ice-orient ed init iat ives,Psychot herapy Research, 25:1, 166-184, DOI: 10.1080/ 10503307.2014.973923

To link to this article: ht t p: / / dx.doi.org/ 10.1080/ 10503307.2014.973923

PLEASE SCROLL DOWN FOR ARTI CLE

Taylor & Francis m akes every effort to ensure the accuracy of all the inform at ion ( the “Content ” ) containedin the publicat ions on our plat form . However, Taylor & Francis, our agents, and our licensors m ake norepresentat ions or warrant ies whatsoever as to the accuracy, com pleteness, or suitability for any purpose of theContent . Any opinions and views expressed in this publicat ion are the opinions and views of the authors, andare not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon andshould be independent ly verified with pr im ary sources of inform at ion. Taylor and Francis shall not be liable forany losses, act ions, claim s, proceedings, dem ands, costs, expenses, dam ages, and other liabilit ies whatsoeveror howsoever caused arising direct ly or indirect ly in connect ion with, in relat ion to or ar ising out of the use ofthe Content .

This art icle m ay be used for research, teaching, and private study purposes. Any substant ial or systemat icreproduct ion, redist r ibut ion, reselling, loan, sub- licensing, system at ic supply, or dist r ibut ion in anyform to anyone is expressly forbidden. Term s & Condit ions of access and use can be found at ht tp: / /www.tandfonline.com / page/ term s-and-condit ions

METHOD PAPER

Building clinicians-researchers partnerships: Lessons from diverse

natural settings and practice-oriented initiatives

LOUIS G. CASTONGUAY1, SOO JEONG YOUN1, HENRY XIAO1,J. CHRISTOPHER MURAN2,3, & JACQUES P. BARBER2

1Department of Psychology, Penn State University, University Park, PA, USA;

2Derner Institute for Advanced Psychological

Studies, Adelphi University, New York, NY, USA &3Department of Psychiatry, Beth Israel Medical Center, New York,

NY, USA

(Received 2 October 2014; revised 2 October 2014; accepted 2 October 2014)

AbstractIn this concluding paper, we identify the type of studies conducted by 11 teams of contributors to a special issue on buildingclinicians-researchers partnerships. Those studies were conducted across a variety of clinical settings. We also integrate thelessons that have emerged from their collaborative initiatives in terms of obstacles faced, strategies adopted to address thesechallenges, benefits gained, and general recommendations offered to facilitate studies conducted with or by clinicians. Thepaper ends with the authors’ thoughts about the future success of practice-oriented research in general.

Keywords: practice-oriented research; practice-based evidence; practice-research network; scientific-practitioner model

The wide gap between science and practice is due inpart to the one-way direction that has mostly definedthe connection between researchers and clinicians(Goldfried et al., 2014); researchers are generatingempirical knowledge with the hope that practitionerswill implement it in their working environment (Kaz-din, 2008). This predominant, top-down approach tothe generation and implementation of empirical know-ledge has no doubt led to major contributions to ourunderstanding and the efficacy of psychotherapy (Cas-tonguay, 2013). However, since it is primarily guidedby the theoretical interest of academicians and fre-quently conducted in highly controlled settings, thistraditional approach to research has not been anoptimal strategy to address day-to-day concerns ofclinicians or to provide easily generalizable (applicable,actionable, and retainable) practice guidelines in clin-ical routine.

In contrast to what may be labeled “evidence-basedresearch” (EBR) stands a bottom-up approach thathas been referred to as Practice-Oriented Research(POR; Castonguay, Barkham, Lutz, & McAleavey,

2013). POR is characterized by studies that are (1)conducted as part of clinical routine, (2) foster theparticipation of clinicians in different aspects of thedecision, design, implementation, and disseminationof research, and (3) allow for the use of collected datain day-to-day practice. POR thus offers opportunitiesfor clinicians to not only contribute to the advance-ment of scientific knowledge but to also be involved insetting up the agenda of future research (Zarin,Pincus, West, & McIntyre, 1997). By relying on theunique expertise and resources of practitioners, itrepresents an antidote to the current state of empiricalimperialism in psychotherapy within which full-timeresearchers have a dominant voice in terms of whatshould be studied and how it should be studied(Castonguay, 2011). Simultaneously, it offers a rem-edy for the colander effect that reflects our inattentionto clinical knowledge and experience (Kazdin, 2008).Rather than being mutually exclusive, EBR and PORcan be viewed as complementary paradigms, wherebythe strengths and limitations (in terms of internal andexternal validity, for instance) of each approach can

Correspondence concerning this article should be addressed to Louis G. Castonguay, Ph.D., Department of Psychology, Penn StateUniversity, University Park, PA 16802, USA. Email: [email protected]

Psychotherapy Research, 2015Vol. 25, No. 1, 166–184, http://dx.doi.org/10.1080/10503307.2014.973923

© 2014 Society for Psychotherapy Research

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

lead to a more comprehensive and robust base ofknowledge (Barkham & Margison, 2007; Barkham,Stiles, Lambert, & Mellor-Clark, 2010).

A variety of POR studies have already been con-ducted (see Castonguay et al., 2013 for a review), butthis type of research is still at an early phase ofdevelopment. In order to generate more interest inand to facilitate future collaboration in studies, 11groups of contributors who have been involved inPOR in different naturalistic settings around theworld were invited to share their experience in aspecial series of papers (Castonguay &Muran, 2014).As a concluding piece, the goal of the current paper isto identify convergences between the clinicians-researchers partnerships featured in this special seriesin terms of studies conducted, obstacles faced, andstrategies used to deal with problems encountered,benefits earned, and general recommendationsoffered, as well as to highlight some aspects that areonly found in particular research programs. We hopethat these common and distinct experiences willprovide useful lessons and guidelines that could behelpful to all clinicians and researchers interested inconducting future POR, as well as offer new perspec-tives to current POR investigators working in theirown respective naturalistic settings.

Studies

A variety of topics have been investigated in the PORinitiatives described in this series. A common focusis the assessment of change using standardizedoutcome monitoring. This involves tracking or pre-dicting the progress of individual, large groups, orspecific types of clients (Adelman, Castonguay,Kraus, & Zack, 2014; Boswell, Kraus, Miller, &Lambert, 2014; Castonguay, Pincus, & McAleavey,2014; Fernández-Álvarez, Gómez, & García, 2014;Holmqvist, Philips, & Barkham, 2014; Koerner &Castonguay, 2014; McAleavey, Lockard, Caston-guay, Hayes, & Locke, 2014; Strauss et al., 2014;West et al., 2014). The use of outcome monitoringin day-to-day practice has allowed some of theseprograms to investigate a number of issues related tospecific patterns of change (such as dose–effect andgood enough models, predictors of differentialresponse patterns, sudden changes, deterioration,and therapist–client responsiveness), as well as theimpact of providing feedback and clinical tools totherapists based on client change.

Several other types of research have been con-ducted by a smaller number of partnerships. Forexample, the effectiveness of psychotherapy has beenassessed, whether conducted in psychological ser-vices or in private practice (Holmqvist et al., 2014;Koerner & Castonguay, 2014). The outcome of

specific types of treatments or interventions for awide range of clinical problems (e.g., children bed-wetting and adolescents with behavior and drugproblems) has also been investigated or compared(Adelman et al., 2014; Fernández-Álvarez et al.,2014; Holmqvist et al., 2014; Szapocznik, Muir,Duff, Schwart, & Brown, 2014). Using differentresearch methodologies (including randomized trialsand single-case experimental designs), studies haveassessed the impact of specific training programsaimed at fostering relationship skills, behavioralactivation techniques, and two-chair techniquesassociated with emotion focused therapy (Koerner& Castonguay, 2014). The differential effectivenessof therapists has also been a core focus of PORstudies (Holmqvist et al., 2014).

Also investigated are a number of characteristicsrelated to the client, such as sociodemographicfactors, treatment history, diagnostic variables, pre-treatment severity and family functioning, andattachment (Adelman et al., 2014; Castonguay et al.,2014; Holmqvist et al., 2014; McAleavey et al.,2014; Szapocznik et al, 2014; West et al., 2014);therapist, such as hours per week of direct care andpersonal style (see Fernández-Álvarez et al., 2014;West et al., 2014); treatment, such as practicesettings, referral process, access to, utilization andprovision of different types of services (psychother-apy and/or pharmacotherapy), sources of paymentand management of care, as well as societal beliefstoward psychotherapy (Fernández-Álvarez et al.,2014; McAleavey et al., 2014; West et al., 2014);and the relationship between some of these variables,such as the congruence between client and thera-pist’s perceptions of symptoms, as well as differencesin diagnosis and treatment provided across patients’race and ethnicity (Holmqvist et al., 2014; Westet al., 2014).

In addition, diverse POR programs have con-ducted process studies, focusing on topics such asthe use of (or adherence/fidelity to) interventionsassociated with empirically supported treatments,consistency of routine care with evidence-basedpractice guidelines, helpful events, therapeutic alli-ance, and principles of change (Adelman et al.,2014; Castonguay et al., 2014; Fernández-Álvarezet al., 2014; Garland & Brookman-Frazee, 2014;Holmqvist et al., 2014; Koerner & Castonguay,2014; Strauss et al., 2014; Szapocznik et al., 2014;West et al., 2014). POR studies have also involvedthe evaluation of assessment measures and DSM-5diagnostic criteria (McAleavey et al., 2014; Westet al., 2014). The development of tools for supervi-sion of evidence-based interventions has also been afocus of a collaborative initiative (Garland & Brook-man-Frazee, 2014). Perhaps reflecting, from an

Psychotherapy Research 167

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

epistemological perspective, an ultimate form ofintegration of science and practice, efforts havebegun to train therapists from different parts ofthe world in designing feasible and highly rigorousresearch (single-case experimental studies) to testhypotheses tied to their clinical practice (Koerner &Castonguay, 2014).

Obstacles

A number of obstacles and difficulties are to beexpected when building clinicians–researchers part-nerships, as well as conducting POR within them. Inthe following text are some of the problems that theauthors in this series have encountered in theircollaborative work.

Clinical Concerns: Is it Worthwhile? Is it

Dangerous? Is it Feasible?

One of the most serious challenges faced by POR isthe fact that the tasks involved can be perceived bytherapists as being irrelevant or even detrimental totheir clinical work. This is a major issue confrontedby the implementation of outcome monitoring sys-tems (Boswell et al., 2014; Fernández-Álvarez et al.,2014; Holmqvist et al., 2014; Strauss et al., 2014).Practitioners are not likely to be fully engaged in datacollection if they are concerned that it might gener-ate negative reactions from clients, create difficultiesin the therapeutic relationship, or simply fail toprovide clinically helpful information. Above andbeyond outcome monitoring, Fernández-Álvarezet al. (2014) argued that any research task can forceclinicians to shift their attention away from anexclusive focus on the immediate clinical situations,to a consideration of the more distal researchimplications of the data collected. As they noted,weighting the long-term value of research data canbe “a challenge to participants’ patience” (p. 8).

Other concerns observed include the fear thatoutcome data might reveal negative findings or thatresults could have negative impact on a clinician’spractice—or a treatment center—in terms of perform-ance evaluation, referrals, or income (Adelman et al.,2014; Boswell et al., 2014; Strauss et al., 2014). Notknowing who will have access to outcome data andwhat will be done with it can also be experienced as amajor threat to clinicians’ autonomy. As evocativelystated by Boswell et al. (2014), clinicians “do not like‘big brother’ and perhaps with good reason” (page 7).Rightly so, practitioners have questioned the ability ofspecific outcome results to accurately capture theclients’ change, as well as the appropriateness ofinterpreting any outcome data without proper contextor consideration of other sources of information (e.g.,

how complex the client’s problems are, including his/her life circumstances; McAleavey et al., 2014;Strauss et al., 2014).

Anxiety and apprehension experienced by clini-cians have not been limited to outcome monitoring.The fear, in the eyes of both therapists and clients, ofpotential breach of confidentiality can be an obstacleto the conduct of any type of research in naturalisticsettings (Boswell et al., 2014; Koerner & Castonguay,2014). As reported by Szapocznik et al. (2014),anxiety can also be raised by videotape observationof sessions (as a means of monitoring therapy adher-ence), or by the adoption of a treatment manual—especially with highly experienced practitioners. Forthe less experienced, the idea of having to ask clientsto participate in research can trigger intense feelingsof impostor syndrome, as some may feel a lack ofcompetence and justification to ask clients to doanything extra for them or for the clinic (Castonguayet al., 2014).

Aside from matters of relevance, immediate value,potential detriment, and anxiety are the issues offeasibility. Research protocols that require too manytasks or intense supervision have been difficult toimplement, let alone adopted as part of routine clinicalpractice after the completion of the study (Koerner &Castonguay, 2014; Szapocznik et al., 2014).

Collaboration and Communication Problems:

Can This Really be a Team?

Not surprisingly, various collaboration and commun-ication problems can jeopardize the design, plan, andimplementation of any kind of POR. First andforemost, researchers must be constantly vigilant ofpotential pitfalls of empirical imperialism. Thesecould manifest by subtle errors of omission, as inhaving innocuous or unplanned conversations aboutstudy design without the presence or previous input ofpractitioners. Or it can take the form of explicitdismissal and exploitation, as when “the researcherdetermines all aspects of the study, agrees with theclinical director to take advantage of the practicesetting’s volume of patients, and then the therapistand client participants are roped into additional workthat may not align with their goals” (Koerner &Castonguay, 2014, p. 9).

Even when true collaboration and active participa-tion has been sought, communication problems areto be expected. With many stakeholders involved,orchestrating the exchange of information representsa difficult endeavor (McAleavey et al., 2014). It is alsoimportant to recognize that stakeholders frequently“talk different languages”: Not only do they have theirrespective jargon, but they can also have discrepantperspectives on the same words. For example, as

168 L. G. Castonguay et al.

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

pointed out by Garland & Brookman-Frazee (2014),a term such as “evidence” can be interpreted invarious ways, evoking very different emotional experi-ences (threat or approval) in those devoting theirprofessional lives to help clients navigate complexlives, in contrast to those paid to contribute to theadvancement of empirical knowledge. Language pro-blems between researchers and administrators havealso been observed regarding the translation ofresearch findings. As noted by McAleavey et al.(2014), these problems can be particularly challen-ging because “questions that seem both criticallyimportant and very simple to administrators areamong the most complex empirical talks available”(p. 9).

Closely linked to variant languages and perspec-tives is the fact that the various partners live indifferent cultures, face different demands and expec-tations, and pursue different goals (Garland & Brook-man-Frazee, 2014; Strauss et al., 2014), all of whichmay at times reflect or lead to conflictual needs—suchas the researchers need to collect publishable data, theclinicians wish to obtain clinically informative data,and the administrators need for actionable data atminimal cost (Boswell et al., 2014). As correctlystated by Garland & Brookman-Frazee (2014), sta-keholders can also operate on “different time tables”(p. 11). For example, clinicians who need to figureout quickly how to address issues that emerge on asession-by-session basis can experience frustration bythe time that it frequently takes for researchers topresent or publish answers to the questions they haveinvestigated together (Castonguay, Nelson et al.,2010). Frustration can also be mutual. As reportedby Adelman et al. (2014), the difficulties involved bythe implementation of a standardized research proto-col within a natural setting can have unfortunateconsequences for time-sensitive academic require-ments (e.g., dissertation projects).

As in any kind of team enterprise, interpersonaldynamics have been identified as challenges toclinician and researcher partnerships. Garland &Brookman-Frazee (2014) noted that “power differ-entials” associated with various professional statusmay influence the way that participants collaborateand communicate. The same authors have alsowarned against the danger of having a partnershipbased on a unidirectional exchange of knowledge, asopposed to a reciprocal one. Intentional or not, thispower dynamic is akin to or is reflecting the issue ofempirical imperialism mentioned earlier. Garland &Brookman-Frazee (2014) have also identified inter-personal problems (personality issues and personalagenda) that, as with any type of group project, caninterfere with POR initiatives.

Pragmatics: Being Bugged Down by Reality

The development, implementation, and mainten-ance of POR also face a number of pragmaticobstacles. For Koerner and Castonguay (2014),practical incompatibilities between research tasksand clinicians’ workflow actually represent the prim-ary challenge of POR. Perhaps the most obvious andintractable of these barriers is time, or lack of it.Irrespective of the world that they live in, most PORstakeholders are extremely busy. And although theyshare an interest in their collaborative endeavor, formany of them such an endeavor represents only afragment of their professional responsibilities. Putbluntly, POR means extra work. For example, inaddition to the training involved in the proper use ofa particular measurement system, outcome monitor-ing requires time to administer and interpret thequestionnaire, provide feedback to clients, and keeptrack of assessment points (Boswell et al., 2014;Holmqvist et al., 2014; McAleavey et al., 2014;Strauss et al., 2014). The more and/or bigger tasksrequired by a research project, the more its prepara-tion and implementation may compete with the dailydemands of all participants involved (Koerner &Castonguay, 2014; Szapocznik et al., 2014; Westet al., 2014). As a case in point, the design of a studyon helpful and hindering events in therapy requiredpractitioners to meet regularly with researchers for ayear. Having to fill out a process measure for each oftheir private clients after every therapy session overthe course of 18-months of implementation alsoforced therapists to sometimes have to choosebetween research tasks and bathroom breaks (Koer-ner & Castonguay, 2014).

Not surprisingly, time for research is particularlydifficult to find when participation is on a voluntarybasis (West et al., 2014). The lack of financialincentives to clients and therapists has indeed beenidentified as an obstacle to the successful imple-mentation of POR (Koerner & Castonguay, 2014).When it applies to outcome monitoring in natural-istic settings, the lack of finances, let alone the costto therapists, reflects an unfair burden. As pointedout by Boswell et al. (2014), whereas physicians donot have to pay for their patient tests, the insuranceindustry has refused to reimburse the routine collec-tion of behavioral health data. Financial support alsorepresents a major source of challenge and stress forlarge naturalistic projects and POR infrastructures(Fernández-Álvarez et al., 2014; Garland & Brook-man-Frazee, 2014; McAleavey et al., 2014; Westet al, 2014). Directly related to the financial needs ofthese large initiatives are the organizational chal-lenges that come with the collection and manage-ment of data across multiple sites, assignment of

Psychotherapy Research 169

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

responsibilities and distribution of resources acrosspartners, as well as training, management, and reten-tion of staff members (Boswell et al., 2014; Fernán-dez-Álvarez et al., 2014; McAleavey et al., 2014;Strauss et al., 2014; West et al., 2014).

Costs: When Research Interferes with Other

Needs

Conducting POR can also be costly for manyinvolved in the partnership. For example, having toremember all tasks required by a research protocol,especially in the early phase of a study, may make itdifficult for therapists to allot their full attention tothe needs of their clients. Moreover, some proce-dures, such as getting informed consent during thefirst session of therapy, can infringe on the thera-pist’s time to conduct a full assessment and/or fostertherapeutic bond (Koerner & Castonguay, 2014).Any research-related tasks, big or small, can alsoimpact practitioners’ (or a treatment center’s) capa-city to generate income. By possibly interfering withcare delivery and earning potential, the time devotedto research can thus be viewed as an unnecessaryluxury by clinicians, clients, and administrators(Adelman et al., 2014). In brief, POR is impededby a double financial challenge (double whammy):Not only there are often no financial incentive forclients or therapists but it also costs them in terms oftime and effort. Researchers also have to strugglewith negative consequences that can come alongwith an engagement in POR, such as the frequentlack of fit between nontraditional research and thepriorities of funding agencies (West et al., 2014), thelack of incentives from academic institutions forcommunity-based work (Garland & Brookman-Fra-zee, 2014), or the incompatibility between thepublishing pressure of academia and the slow paceand the demanding efforts of practitioners andresearchers collaborations (Koerner & Castonguay,2014). Accordingly, researchers interested in build-ing POR initiatives should consider inviting othersliving in their world (such as colleagues and stu-dents) and seeking administrative assistance—espe-cially if they are not yet tenured (Castonguay,Nelson et al., 2010; Castonguay et al., 2014).

Fostering Strategies

A number of strategies have been identified by thecontributors of this series, both to address theobstacles mentioned earlier, as well as to facilitatePOR studies and partnerships in general.

Putting Premium on Clinical Relevance and

Beyond

It should be of no surprise to anyone that studiesthat fail to be directly related to clinical practice arenot likely to stimulate therapists’ engagement. Oneof the lessons that emerged from POR collabora-tions, however, is that in order to generate andmaintain practitioners’ commitment, such studieshave to go beyond the threshold of “clinical relev-ance” and be more than potentially interesting; theyhave to suggest ways of improving clinical work.“Clinicians will help with research—if the researchhelps them clinically” (McAleavey et al., 2014, p. 6)is one of the major conclusions derived from a surveythat asked busy counselors what kind of researchthey would be willing to participate in, knowing thatthis participation would require additional work andtime on their part.

One way to increase the helpfulness of research isto integrate it into different aspects of clinical work.For example, Fernández-Álvarez et al. (2014) poin-ted out that using data within clinical supervisioncan increase therapists’ motivation to collect it.Perhaps the ultimate test of helpfulness is thatstudies have to be actionable and retainable. Tofacilitate the clinician’s willingness to go along withresearch tasks and cope with the added stress andanxiety that may come with them, these tasks have tobe immediately informative by providing therapists,for example, “here-and-now” guidance about inter-ventions that can be used to best address the clients’needs (Koerner & Castonguay, 2014). RepeatingFernández-Álvarez et al. (2014)’ s wise comment, itis testing the therapist’s patience to present findingsonly after completion of a study. As argued else-where, research tasks are likely to be performedduring the study (and retained in clinical routineafter its completion) if they are “clinically syntonic”(Castonguay et al., 2010). Beyond the abstractconcept of “clinical relevance,” POR investigationswill be most successful if they foster a seamlessintegration of research and practice or, put differ-ently, if the study protocols confound research andpractice. As described elsewhere, “clinicians trulyintegrate science and practice every time they per-form a task in their clinical practices and are not ableto provide an unambiguous answer to questions suchas: ‘Right now, am I gathering clinical informationor am I collecting data?,’ or ‘At this moment, am Itrying to apply a helpful intervention with my clientor am I implementing a research task?’” Frequently,setting up rigorous empirical investigations will leadthem to answer these questions by saying, “Perhapsboth,” may be the most fruitful and exciting pathwayto bridge research and practice (Castonguay et al.,

170 L. G. Castonguay et al.

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

2010, p. 352). Koerner & Castonguay (2014) alsouse the metaphor of research and practice beingwoven as a whole cloth to describe POR studies thatcause minimal disruption to the clinical workflowand that may “improve clients outcome by meetingtherapists’ learning needs” (p. 4). It should also beobvious that in many cases confounding researchand practice can be a fruitful strategy to protect thelimited resource of time. The more infringement ontime, the less likely it is to learn empirically fromclinical practice.

Also going beyond the concept of relevance,clinicians are more likely to participate in studiesand continue to use research procedures in poststudycompletion when such procedures do not requiredrastic changes to their practice (Castonguay, 2013;Koerner &Castonguay, 2014).Most clinicians will beindifferent to or even resentful if they are asked to putaside the way they have been practicing for years andforced to adopt completely new treatment packages(and/or theoretical orientations) in order to test aprotocol within their own clinical routine. Rather,they are more likely to join a project and retain whatthey might learn from it if what they do for research isadditive to their clinical repertoires, such as obtainingprocess and outcome feedback from their clients,using new interventions to address specific clinicalissues, or having access to clinical tools that they canuse on their own terms and time schedule (seeAdelman et al., 2014; Koerner & Castonguay, 2014;McAleavey et al., 2014).

In POR, or any kind of psychotherapy research,clinical relevance is frequently brought up in contrastwith scientific rigor—as if clinical utility and internalvalidity were opposite poles of a continuum, orirreconcilable categories by which one is to judgeresearch quality. It should indeed be recognized thatthe strengths of POR, such as high external validity,may at times come at a price. As noted in Koerner andCastonguay (2014), some methodological compo-nents of psychotherapy research that can increase itsinternal validity (e.g., multiple observer assessmentsof pre- and posttreatment outcome) are not likelyfeasible or desirable in most studies conducted inclinical routine. However, it is also important to avoidfalse dichotomies. First, naturalistic studies can reachboth high levels of clinical helpfulness and validity,internal and external. This has been illustrated by theuse of randomized clinical trials and single experi-mental designs in several studies presented in thisseries (Boswell et al., 2014; Koerner & Castonguay,2014; Strauss et al., 2014; Szapocznik et al., 2014).More importantly, rigor and relevance can be seen ascomplementing and reinforcing of each other (Gar-land & Brookman-Frazee, 2014). For example, the

more valid an instrument or a finding is, the moreconfidence wemay have that they are truly actionable.

Yet, it is undeniable that a tension frequently existsbetweenmaking a study both feasible and scientificallyrigorous. At least two directions have been suggestedin this special series with regard to finding a balancebetween these crucial issues. The first is to offer someflexibility (or customization) in the way that instru-ments can be used and data can be collected. Forexample, in the large practice-research network(PRN) infrastructure of university counseling centersdescribed by McAleavey et al. (2014), specific siteshave the ability to change the order and turn on or offindividual items of one of the standardized measuresused by all participating centers. Another strategy is toexplicitly recognize that no study can ever be perfectand that partners have to make an informed choice interms of level of rigor balanced with the costs entailed(Koerner & Castonguay, 2014).

Addressing Threats and Anxiety

As described earlier, fears of negative impact (e.g.,breach of confidentiality, threats of autonomy, risks ofnegative evaluation, and potential decreases in refer-rals and revenue) and concerns about the clinicalaccuracy of empirical data represent major challengesin building up clinicians–researchers partnerships.For many of those involved in POR, a key componentto address these understandable apprehensions istransparency (Boswell et al., 2014; Strauss et al.,2014). For example, Boswell et al. (2014) haverecommended a full disclosure in writing regardingissues such as confidentiality. Researchers have alsofound that clinician participation is enhanced not onlywhen they are informed of the complete anonymity ofthe data collected but also by the explicit reassurancethat the data will not be used to control the financingof their practice or to replace clinical judgment(Strauss et al., 2014). POR researchers have clearlyvoiced that empirical data, even collected in thenaturalistic setting where it is used, are not sufficientto guide clinicians about what to do, when to do it,and with whom. In fact, we tend to forget that thephilosopher David Hume (1739) had already men-tioned a few centuries ago that one cannot get frommaking descriptive statements (“is”) to making pre-scriptive statements (“ought”). Not only should databe interpreted within the context of the client’s lifeand current situation, but it should also be viewed ascomplementary to clinicians’ judgment and used topoint to potential directions for further professionaldevelopment and training (Castonguay et al., 2013;Holmqvist et al., 2014; McAleavey et al., 2014).

In addition to recognizing the limitations of theirinstruments, some researchers have also stressed that

Psychotherapy Research 171

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

one way to address clinicians’ concerns about theaccuracy and adequacy of empirical data is tocontinually seek to increase the psychometric qualityof measures, improve their utility and predictability(e.g., what types of clients are particular therapistspredominantly effective with?), and provide clinicaltools (Boswell et al., 2014).

Another apprehension experienced by clinicians iswhether the research tasks can be of value to theirclients. Data can be helpful in decreasing such fear. Acase in point is how outcome data were used toaddress the imposter syndrome experienced bygraduate students which, as mentioned earlier, madethem anxious to ask anything of their clients, includ-ing filling out pre and posttreatment measures orparticipating in studies conducted by their peers.Showing students that compared to experiencedclinicians in private practice, their interventions hadhigher impact on serious clinical difficulties such assuicide, and psychosis was a turning point in terms oftheir sense of self-efficacy and motivation to use andconduct POR studies (Castonguay et al., 2014).Szapocznik et al. (2014) have also observed that thepresentation of empirical evidence about the efficacyof a treatment to be tested can facilitate variousstakeholders’ buy-in.

Yet having clinicians share their experiences withother clinicians might be an even stronger strategyto address apprehensions, as it avoids perceptions ofbias, self-serving, or controlling motivation fromresearchers. As Boswell et al. (2014) have learnedfrom their experience:

researcher’s attempts to impart the “wisdom of rou-tine outcome monitoring” are far less effective thanthe wisdom imparted by fellow clinicians who haveused the particular outcome monitoring system ofinterest. It is through direct clinical experience and bysharing these experiences (e.g., through vignettes)that other clinicians begin to seriously entertain thepotential benefits. (p. 8)

It is also important to note that such direct experienceis frequently discordant from previous expectations ofclinicians who are being asked to use outcomemonitor-ing. As such, perception of the relevance and value of aninstrument is sometimes acquired via a correctiveexperience (Youn, Kraus, & Castonguay, 2012).

While the strategies mentioned above can andshould be used to address clinicians’ concerns abouttheir participation in POR, it is, nevertheless, crucial toconstantly gather feedback and closely attend to thera-pists’ criticisms about the protocols (assessment, treat-ment, or otherwise) implemented (McAleavey et al.,2014; Szapocznik et al., 2014). This will not only builda stronger sense of collaboration but is likely to alsoimprove the quality of the research conducted.

Pumping Blood in the Partnership: Enhancing

Communication and Collaboration

For it to be worthwhile, a partnership should bebased on a diversity of expertise and opinions. Trueadvances in complex fields are rarely achieved bythe joint work of individuals who think the same way.As cogently stated by Garland and Brookman-Frazee(2014), partnership members “should possess com-plementary, but non-redundant knowledge andexperiences that can be combined and contextualizedto facilitate knowledge creation and innovation” (p. 6).Like any kind of teamwork, however, the success ofPOR also rests on strong communication and collab-oration between individuals who typically live indifferent professional worlds. Metaphorically, com-munication and collaboration is the blood that main-tains the life in professional partnerships. As describedearlier, joint research initiatives face serious chal-lenges, including different languages, perspectives,goals, expectations, demands, as well as wishes andfears of various stakeholders involved. These discrep-ancies are unavoidable and should be faced withtransparency, as well as frequent and open dialogues.

Many papers in this series have emphasized theimportance of regular meetings to discuss, under-stand, validate, and optimally incorporate the needs,concerns, and contributions of diverse collaborators,in addition to remind or inform all parties involvedof the goals, tasks, and progress of the joint projects(Boswell et al., 2014; Castonguay et al., 2014;Fernández-Álvarez et al., 2014; Garland & Brook-man-Frazee, 2014; Koerner & Castonguay, 2014;McAleavey et al., 2014; Strauss et al., 2014). Asnoted by Garland & Brookman-Frazee (2014), face-to-face meetings are crucial to build trust anddevelop a shared language. For these authors, suchtrust and common language are dependent upon anumber of interpersonal processes that are commonto many successful relationships, including an open-ness and responsiveness to others’ perspectives, areciprocal enthusiasm about the collaboration and,interestingly, a willingness “to go above and beyondan agreed upon scope of work” (p 7). Yet, Garland &Brookman-Frazee (2014) have also pointed out thatpartners should be aware of potential pitfalls that canundermine face-to-face meetings, such as the failureto recognize and adequately process invalidatingpower differentials, unidirectional sharing of know-ledge, and monopolization of control. To prevent ordeal with these group processes, they argued it is“important to address members’ expected roles andunique contributions, and the distribution of powerat the outset, as well as explicitly establishing normsfor working together” (p. 8). At the same time, theywisely recommend that input from all members

172 L. G. Castonguay et al.

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

should not only be attended to but should alsocontribute to actual changes in collaborative work.Not surprisingly, members of POR have also indi-cated that interpersonal problems, in the form ofconflict and alliance ruptures, are to be expected andshould be addressed to foster the development andoptimal functioning of collaborative partnerships(Castonguay et al., 2014; Garland & Brookman-Frazee, 2014)

Fundamentally, successful POR requires twothings: A strong level of engagement and commitmentfrom each member to the philosophy and tasks ofresearch (Holmqvist et al., 2014; Strauss et al., 2014),as well as a genuine alliance between them. This stateof alliance both reflects and fosters a sense of equalityand respect, as well as a recognition of diverse ways ofunderstanding and investigating complex realities(Castonguay et al., 2013). The same state of allianceor collaboration has been referred by Garland &Brookman-Frazee (2014) as “egalitarian leadership.”This overarching principle of healthy group process,however, does not negate differential skills and theadaptive strategy of matching leading responsibilitieswith specific knowledge and expertise. But whategalitarian leadership implies, process and outcomewise, is full rights of all participants. For clinicians, thismeans that they should have access and control overwhat is frequently in the researchers’ exclusive prov-ince. As Holmqvist et al. (2014) noted, “a key processthat needs to be achieved in practice-based evidence isto ensure a sense of local ownership by practitioners inthe data they collect” (p. 8). This includes providingopportunities for clinicians to be involved in the plan(not only with respect to what to analyze but also whatnot to analyze; see Strauss et al., 2014) and conduct ofstatistical analyses of data collected, as well as torequest data for the investigation of questions relatedto their own interest (Koerner & Castonguay, 2014;McAleavey et al., 2014). Increasing a sense of owner-ship can also be achieved by involving practitioners inthe selection of instruments to be used for researchpurposes (Holmqvist et al., 2014) and by giving themthe opportunity to modify the way these measures canbe used. An example of this is the decision of therapistsin training to improve the clinical utility of outcomemonitoring by implementing it, as part of their PORinfrastructure, after every session rather than at limitedassessment points (Castonguay et al., 2014).

As illustrated in several papers in this series,collaboration in POR studies can involve a broadarray of stakeholders (e.g., therapists, supervisors,researchers, graduate students, administrators at dif-ferent levels of management, funders, parents, judges,and policy decision-makers), each of them deservingto have their voices heard and their expertise recog-nized in the research partnership (see Adelman et al.,

2014; Boswell, et al., 2014; Garland & Brookman-Frazee, 2014; Koerner & Castonguay, 2014; McA-leavey et al., 2014; Strauss et al., 2014; Szapoczniket al., 2014; West et al., 2014). Building upon suchdiversity of partners can not only strengthen a studybut also help generate “valued research findings thatmay be more likely to have an impact on clinicalpractice and policy” (West et al., 2014, p. 12). Andwhile it should be recognized, as we did before, thatall stakeholders have different needs, expectations,and demands, it is also important to keep in mind thatthese diverse goals can be highly complementary.There is, for example, no a priori counterindicationbetween clinicians’ desire to get quick and helpfulinformation about the pattern of change of an indi-vidual client, researchers’ interest in identifying mod-erators and mediators of change in groups of clients,and administrators and funders’ hope of finding waysto efficiently address the needs of all their clients.These are different levels of investigation that canhave synergetic impact on each other, where theanswer to each particular question can benefit fromand contribute to the clarification of other questions.

As illustrated in several papers in this series,partnerships can also be built upon and enhancedby focusing on convergent goals. Academicians andclinicians can actually be pleasantly surprised at thehigh level of convergence in their interest andresearch ideas (Adelman et al., 2014; Castonguayin Lampropoulos et al., 2002). Considering howpowerful a motivation such shared interest could be,it is not surprising that many authors of this serieshave emphasized the value of clinicians’ full involve-ment in the selection of the ideas to be examined,design to investigate them, recruitment of partici-pants, implementation of the study, and efforts todisseminate findings (Koerner & Castonguay, 2014;Szapocznik et al., 2014; West et al., 2014). Stake-holders also share general goals, above and beyondthe specific focus of a particular study. At least twomajor ones can be delineated: Many individuals(including, of course, clinicians) are interested incontributing to the advancement of empirical know-ledge and the reduction of the gap between scienceand practice, and most, if not all, professionals in themental health field are invested in improving the careof clients. Accordingly, one way to foster partner-ships is to lead stakeholders to identify themselves,both at a personal and an organizational level, withresearch projects that are specifically aimed at thesefar-reaching goals (Fernández-Álvarez et al., 2014).Put differently, successful POR can be fostered bybuilding a “sense of community” (McAleavey et al.,2014) that is guided by the shared ambition tocontribute to the advancement of knowledge andreduction of suffering.

Psychotherapy Research 173

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

Making it Possible: Resources and Pragmatics

McAleavey et al. (2014) have identified two funda-mental resources for the development and mainten-ance of their large PRN infrastructure, and it is fair tosay that these supplies are necessary for any kind ofPOR initiative: “Time (and a lot of it) and people (anda lot of them)” (p. 16). We have already mentionedthe large number and variety of stakeholders involvedin these partnerships. What has also been emphasizedby several authors in this series is the amount of timethat collaborative work can require with respect to thepreparation, coordination, and implementation of astudy (Garland & Brookman-Frazee, 2014; Koerner& Castonguay, 2014; Strauss et al., 2014; Szapoczniket al., 2014; West et al., 2014). For example, thepreparation of one of these studies (which includedthe development of the research design, planning ofthe management, analysis and publication of the data,and the allocation of funding) took four years (Strausset al., 2014).

A particular aspect of preparation that has beenhighly recommended might best be called the “routi-nization” of a protocol. Routinization begins withthorough planning. Clinicians and researchers havelearned that the more time you spend laying down thedetails of a protocol and anticipating problems thatcould emerge when implementing it, the more yousave in terms of energy, frustration, and time later(Koerner & Castonguay, 2014). Specific strategieshave been developed to help practitioners “automat-ize” (learn, remember, and recall) the research pro-cedures as part of their clinical routine, such asthrough the use of web technology (e.g., Krug’s(2006) “Don’t make me think”), or the constructionof multiple scripts, each of them including the sameresearch procedures but with differing levels ofdetails. Additional recommendations that have beenmade to facilitate “routinization” are the inclusion ofemail and phone consultations, as well as frequentmeetings at the beginning of a study to discussproblems that some participants have faced and waysthat other partners have devised to prevent andresolve difficulties. It also seems advisable to haveclinicians implement the research protocol, or at leastparts of it (e.g., core measures), in their clinicalroutine before the study is launched (Koerner &Castonguay, 2014; Strauss et al., 2014). Such pilotwork not only provides opportunities for practice andconsultation but also sets up an optimal test forparticipants to decide whether or not the benefits ofresearch procedures (e.g., in terms of actionableinformation they can provide during treatment) out-weigh their costs (e.g., in terms of disruption ofclinical workflow).

Another concrete key to successful POR is tokeep things as simple as possible and to avoidimposing unreasonable burden to clinical routine(Boswell et al., 2014; Koerner & Castonguay, 2014;West et al., 2014). Based on their separate experi-ence, Koerner & Castonguay (2014) concluded that“[t]he best strategy we have found in our practice-oriented research designs is to accept the constraintsfaced by practitioners and design research proce-dures that map as directly onto clinical care alreadyprovided as possible” (p. 9). The clinicians’ con-tribution in the design of feasible studies is critical—as they know best what is possible and impossible toadd to their day-to-day work schedule (West et al.,2014). A good example of such wisdom comes fromone of the lessons learned in a PRN study con-ducted in private practice (see Koerner & Caston-guay, 2014). In this study, clinicians andresearchers had decided that each therapist wouldbe inviting all of their new clients to participate.Because this study required therapists to fill outmeasures at the end of every therapy session, itbecame clear that having their entire client caseloadas participants was too burdensome. Based on thisexperience, the subsequent study conducted in thesame PRN involved no more than four participatingclients for each of the therapist at any given point inthe study.

Feasibility, however, not only refers to how possibleit is for clinicians to adopt and adequately implementa research protocol but also how to sustain suchimplementation. Organizational support has beenidentified as a critical addition at each of these steps(Koerner & Castonguay, 2014; Szapocznik et al.,2014). For example, Szapocznik et al. (2014) colla-borated with treatment agencies to recruit “on-site”supervisors for the training, monitoring, coaching,and provision of feedback to clinicians participating ina psychotherapy effectiveness study. In another study,clinicians greatly benefitted from the help of researchassistants (graduate and undergraduate students),who kept a close and timely monitoring of datacollected, providing them with rapid feedback aboutadherence problems observed, and were easily reach-able to answer questions regarding the study protocol(Koerner & Castonguay, 2014). Garland & Brook-man-Frazee (2014) have argued that, ideally, organ-izational support should not be tied to one specificstakeholder, and instead, can be shared among vari-ous members, such as between administrative staffand university research assistants (Koerner & Cas-tonguay, 2014). In other initiatives, however, theadministrative support has been provided primarilyby the research stakeholders, via grant funding oruniversity funds (e.g., Garland & Brookman-Frazee,2014; McAleavey et al., 2014). Irrespective of its

174 L. G. Castonguay et al.

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

source, POR are likely to be particularly burdensome,in terms of time and energy, if administrative help isnot provided to both clinicians and researchers.

The provision of concrete incentives has beenrecommended, including financial rewards (forboth clients and therapists), as well as credits towardprofessional licenses (Boswell et al., 2014; Koerner& Castonguay, 2014; Strauss et al., 2014; West et al.,2014). Successful conduct of POR, small and large,has also benefited from continued refinement ofcomputer and online technology—to train partici-pants, implement protocols, as well as to collect andmanage data (Boswell et al., 2014; Castonguay et al.,2014; Koerner & Castonguay, 2014; McAleaveyet al., 2014; Strauss et al., 2014; West et al., 2014).The technical challenges involved for the efficientadministration of instruments and management ofdata has at times been met by integrating softwarecompanies as part of the POR partnership (Caston-guay et al., 2014; Koerner & Castonguay, 2014;McAleavey et al., 2014). Others have built their owntechnology to collect data in naturalistic settings (seeFernández-Álvarez et al., 2014; Koerner & Caston-guay, 2014).

Technology can, of course, be expensive. Needlessto say, this is not the only cost entailed by research.And, of course, the larger a study is, the moresubstantial are the financial needs. But since it hasbeen observed that monetary incentives can facilitatedata collection (Koerner & Castonguay, 2014; Westet al., 2014), funding is an important pragmatic issuefor any type of POR, small or large. Several investiga-tors have been successful in securing substantialfinancial support (e.g., Garland & Brookman-Frazee,2014; Holmqvist et al., 2014; Koerner & Castonguay,2014; Strauss et al., 2014; Szapocznik et al., 2014;West et al., 2014). However, it is also a fact that formental health investigators, and perhaps for psycho-therapy researchers in particular, external funding isextremely difficult to obtain. Some POR programshave benefitted from support outside of much prizedfunding sources (mostly governmental), includingprivate foundations, professional associations, anduniversity or treatment center internal funds (e.g.,Adelman et al., 2014; Castonguay et al., 2014;Koerner & Castonguay, 2014; McAleavey et al.,2014; West et al., 2014). As a nonprofit researchinfrastructure, theCenter forCollegiateMentalHealth(CCMH) has also been able to rely on membershipfees from its participating counseling centers, as well asfrom individuals and companies interested in itsintellectual properties (McAleavey et al., 2014).

It may well be, however, that the financial founda-tion of most POR, especially if partners are interes-ted in long-term sustainability, has to rely in part onthe concept of “patching.” As defined by Garland &

Brookman-Frazee (2014), patching refers to thereorganization of partnership and donation ofresources when there is no external funding. Infact, several of the POR initiatives described in thisseries have operated within a “pre-patching” mode,i.e., without having had any or enough externalfunding to fully support their research activities.Instead, they have received contributions from busi-ness partners (e.g., software companies), donation oftime (from therapists, students, and researchers),and even funding from their own members (e.g.,Fernández-Álvarez et al., 2014; Koerner & Caston-guay, 2014; McAleavey et al., 2014). This level ofparticipatory process clearly demonstrates a strongcommitment toward two major goals mentionedabove (contribution to the advancement of know-ledge and the improvement of mental health care),but it also reflects the synergetic and meaningfulcontribution that can be generated from a milieu thatis characterized by mutual trust and a shared pursuitof professional actualization. As stated by Fernán-dez-Álvarez et al. (2014), clinicians set aside per-sonal resources because they know that the conductof research fits institutional needs, which in turn “areoriented to meet the individual’s professional devel-opment” (p. 8).

Though there is no doubt that financial supportcan be extremely beneficial and even crucial for largePOR initiatives, it should also be considered thatexternal funding may, in some circumstances,become a curse. In the current context of “get grantsor perish,” funding might be the principal motivationfor some academicians to establish connections withclinicians. At worst, using clinicians’ time and milieuonly to please a dean or a chair would be committinga faux pas that is beyond empirical imperialism—itmay well be nothing less than “empirical invasion.”At best, the search for funding for the sake offunding is likely to guarantee that a research pro-gram, including the long-term implementation of itsfindings, will cease to continue once the grant endsand the researcher will look for other “hot” fundingareas. Once the research team begins to pay any-body, it is hard to “go back” to a place wherecollaborators are not paid. If we want to conductstudies that lead to retainable findings, we shouldtherefore strive to avoid becoming dependent onexternal findings, at least in some contexts.

Handling Organizational Challenges

As mentioned earlier, true and successful partner-ship is based on transparent and open communica-tion. In the case of large, including multi-sites,collaborations, however, another layer of commun-ication must be addressed: The orchestration and

Psychotherapy Research 175

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

dissemination of information to various partners.Different tools have been used to facilitate thiscrucial aspect of organizational functioning, includ-ing email lists to raise and address problems betweensites, web-based methods to facilitate communica-tion (e.g., google groups) or to submit researchprojects to a centralized research team, in-personand video (webinars) training, annual meetings withrepresentatives of local sites, research conferences topresent projects conducted across the collaborativeinfrastructure, and in-print publications (e.g., annualreports) to describe research findings in friendly (forboth clinicians and administrators) ways (Garland &Brookman-Frazee, 2014; McAleavey et al., 2014).

Another organizational challenge that must be metby large POR initiatives is the centralization of data, inorder to minimize data sources and reduce resourcesneeded to manage it (Strauss et al., 2014). As notedby Strauss et al. (2014), procedures to ensure qualityof data have to be systematically assessed andimproved upon. A good example of the complexityrequired for quality control is provided by the stand-ardization and centralization process developed forthe CCMH PRN, which aimed at ensuring that:

the materials reach clients in the same format everytime; that themeasures are easily administered by andmeet the needs of each UCC; that the data fromclients are efficiently and accurately recorded, scored,reported to the counselor and transmitted to CCMH;and that any future updates to the standardizedmaterials can be accommodated. (McAleavey et al.,2014, p. 7)

The goal of centralizing data collection is particu-larly challenging and requires complex technologicalskills when data is collected and stored using differ-ent software packages, as it is the case for CCMH.

Centralized coordination, however, is not restrictedto data management. Recommendations for thesuccessful operation of multisite initiatives have alsoemphasized the coordination of the various aspectsrelated to a study. For example, one of the mostbaffling tasks of research, for most clinicians andadministrators, is the submission of proposals to getethical approval for conducting a study. In CCMH,for instance, such time-consuming requirement hasbeen handled by having a team of graduate students(themselves part of the centralized research team) toprovide examples of and feedback to the various sitesin submitting their respective research ethic proposal.

To facilitate the aforementioned tasks of commun-ication, data management, and project coordination,many large POR initiatives have created advisoryboards (e.g., McAleavey et al., 2014; Strauss et al.,2014; West et al., 2014), which bear the broaderresponsibilities of providing recommendations about

research agenda and potential sources of funding, aswell as ensure that current and future projects aresensitive to the needs of different stakeholders andconsistent with ethical standards.

While a centralization process and administrativeboard structures might be an efficient way to overseeand manage large projects, the implementation ofsuch projects generally takes place at specific sites.To increase the probability of such implementation,some POR programs have recommended the identi-fication of “local champions” (Boswell et al., 2014),“study champions” (Garland & Brookman-Frazee,2014), or “model managers” (Szapocznik et al.,2014). These are individuals responsible for buildingtrust with stakeholders’ on-site, easing the adoptionand implementation of research protocols, helping toadjust the project to be consistent with clinicalroutine, providing training with regard to tasks,expectations, and anticipated benefits, as well as tomonitor and facilitate the data collection.

Whether it is accomplished by one specific indi-vidual on-site or a group of representatives of differentstakeholders, a key organizational task is to preserve acontinuity of information regarding goals, proce-dures, and problems faced in POR (Boswell et al.,2014; Strauss et al., 2014). This is an especiallycrucial issue when recruitment of new participantsand/or turnover of staff members are anticipated. Theprinciple underlying this recommendation is that“you can never communicate too much” (Castonguayet al., 2014, p. 10). As an example, the first author ofthis paper and his colleagues observed that while thestudents who had been involved in the developmentof the PRN within our training clinic were fullycognizant of the purposes and benefits of combiningthe research and clinical requirements of the doctoralprogram, this was not always the case for later cohortsof students. Accordingly, many from this latter groupfelt that procedures implemented by former studentswere additional burdens imposed on their alreadyhectic clinical responsibilities. To address this obvi-ous obstacle, annual meetings are organized byfaculty, clinical staff, and advanced graduate studentsto describe the origin and goals of our PRN, as well asto inspire a sense of ownership of the data collected atthe clinic. As noted in Castonguay et al. (2014), thesemeetings are aimed at conveying one message: “Thisis not for us (faculty members), and not imposed byus. It is mostly for you and it has been driven in part byprevious and current students” (p. 11)

Building large and long-standing partnerships canalso involve organizational tasks or procedures thatare foreign to the daily activities of most cliniciansand researchers. As noted by McAleavey et al.(2014), for example, “large-scale PRNs are verylikely to include the use of intellectual property or

176 L. G. Castonguay et al.

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

the development of corporate partnerships, and ourexperience suggests planning for risk-managementand legal resources is necessary for success” (p. 10).As another case in point, the transformation of atraining clinic into a fully operational PRN requireda unique agreement with the office of researchprotection, which itself involved extensive negoti-ation with leaders of this office and their attorney(Castonguay et al., 2014).

Researching Research and Processing Process

Two other strategies, or meta-strategies, have beenrecommended to foster POR.One is to rely on researchthat has been conducted on research collaborations.For example, Garland & Brookman-Frazee (2014)have derived lessons from management and imple-mentation sciences, as well as from studies on factorsof successful research–practice partnerships in publichealth, to construct a model of research–communitypractice partnership; a model which has guided thedevelopment of their own researchers and clinicianscollaboration.Of course, research directly related to thecollection and use of psychotherapy data in naturalisticsettings should be highly encouraged. As an example,Boswell et al. (2014) argued that research is sorely“needed on the factors (e.g., participant factors, organ-izational factors, training factors) that facilitate orinhibit the adoption, implementation, and sustainabil-ity of routine outcome monitoring” (p. 11).

Just as the research on research implementationmight facilitate the conduct and use of POR, so is theuse of conceptual models of psychotherapy process tounderstand and improve the process of research inpsychotherapy. This is a conclusion that can bederived from the experience reported by Szapoczniket al. (2014), who have used their family therapymodel to guide the testing of the same model innaturalistic settings. Put in other words, they relied onprinciples underlying their conceptual framework oftherapy to anticipate and deal with organizationaldifficulties faced in their research program. Byaddressing difficulties encountered during three dif-ferent phases of research–practice partnership (adop-tion, fidelity, and sustainability), their work has notonly been consistent with major recommendations ofimplementation sciences, it has also offered aninnovative contribution to this literature. Reflectingan intrinsic integration of science and practice, therecognition of the potential benefits of “using a modelto test a model” was facilitated by the fact that theresearchers involved in their POR were clinicallytrained. There is no reason to assume that their model(or any other theoretical frameworks of therapy) couldnot provide insightful recommendations, process and

content wise, about research on various aspects ofpsychotherapy in clinical routine.

Benefits

Successfully resolving major challenges that comewhen building professional partnerships is, of course,intrinsically gratifying. In addition to such trans-cending reward, POR can have benefits for each ofits stakeholders and beyond.

Improving Clients’ Outcome

To be viable, ultimately, POR has to be beneficial toclients. As a paradigm of applied science, its credib-ility rests in part on its ability to have an effect ontreatment outcome. Based on a diversity of methodo-logies, including randomized clinical trials, there isevidence that research collaboration between clini-cians and researchers can impact psychotherapy inclinical routine (e.g., Adelman et al., 2014; Szapocz-nik et al, 2014). Related to the issue of outcome, thispartnership can also lead to an increase of treatmentretention compared to usual clinical care (Szapoczniket al., 2014). Moreover, POR findings have demon-strated that feedback on progress (as well as theprovision of related clinical tools) can significantlyreduce the rate of deterioration in psychotherapy(Boswell et al., 2014; Lambert, 2010). While itremains to be seen if it is beneficial across diagnosticgroups and settings (e.g., Johnson, 2014), outcomefeedback is providing tools for clinicians to meet theirmost important ethical responsibility, “first do noharm.” Outcome monitoring and feedback can alsoimprove the cost effectiveness of psychotherapy; whentherapists are receiving feedback on therapeuticchange, patients who show early improvement haveshorter treatment durations than those who do not(Strauss et al., 2014).

While more traditional research has focused exten-sively on the impact of particular forms of therapies,POR studies have shown that client outcome are inpart due to the individual therapist they are seeing.Specifically, clients seen by particularly effectivetherapists have a higher probability of being betteroff at the end of treatment than those who are seen byparticularly ineffective therapists (Castonguay et al.,2013). Evidence emerging from POR also suggeststhat particular therapists may have specific areas ofoutcome expertise, fostering some types of change (e.g., reduction of depression) more than others (e.g.,reduction of substance-abuse symptoms; Kraus, Cas-tonguay, Boswell, Nordberg, & Hayes, 2011). Ifappropriately used in clinical routine, such findingson outcome variability (between and within thera-pists) can be a valuable source of feedback about a

Psychotherapy Research 177

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

practitioner’s unique strengths and limitations(Strauss et al., 2014)— which should ultimately bebeneficial to their clients.

Enhancing the Therapeutic Process

Empirical data collected by clinicians also has thepotential of facilitating the process of therapy. Out-come monitoring, for example, can help case for-mulation by providing assessment of a range ofproblems at different phases of treatment (Boswellet al., 2014; McAleavey et al., 2014), at timesrevealing difficulties that clients are reluctant toshare verbally or that therapists may not havethought to ask. Tracking outcome can thus opencommunication about needs—told or untold, met orunmet. It can also help treatment planning byanticipating patterns of change, with regard to clientswho are likely to change, those who are not (Cas-tonguay et al., 2014), and those who are at risk ofdeterioration (Boswell et al., 2014; Lambert, 2010).Since research indicates that clinicians are generallynot good at accurately predicting clients who willdeteriorate during treatment, such empiricalinformation can be invaluable to shift the focus oftreatment as needed (Boswell et al., 2014; Lambert,2010). In contrast, as reported by Strauss et al.(2014), clients tend to have a positive view ofoutcome monitoring, which by itself, can contributeto the quality of the therapeutic relationship, and forthose clients who are progressing well, the presenta-tion of data documenting their improvement canreinforce the working alliance (Boswell et al., 2014).

Helpful feedback is not restricted to outcomeimprovement, or lack of thereof. For example, clients’descriptions of helpful and hindering events duringtherapy can provide therapists with unique informa-tion that might help them adjust their interventions tobetter address their clients’ needs (Koerner & Cas-tonguay, 2014). Interestingly, asking clients to identi-fy such events at the end of every session can providethem with an opportunity to reflect on and processtheir therapeutic experience. For some of them,writing down positive and negative aspects of treat-ment is an easier way to provide feedback thanverbally expressing them (Koerner & Castonguay,2014). As described by Fernández-Álvarez et al.(2014), a variety of data collected as part of theclinical routine (e.g., notes, video tapes, and assess-ment measures) can help detect difficulty in thetreatment process and provide guidance for modifica-tions of treatment (e.g., adding family therapy,enhancing involvement of client’s social supportnetwork, and adjusting frequency of sessions).

At a scientific level, some POR studies haveprovided findings that contribute to our

understanding of the process of change. For example,Szapocznik et al. (2014), found that therapist’sadherence to theoretically specific components oftheir family-based treatment for substance-abuseadolescents was associated with higher retention,greater engagement, as well as better outcome interms of family functioning and substance use in theadolescent clients. In another study conducted withtherapists of different theoretical orientations, inter-ventions intended to increase awareness were per-ceived, by both clients and therapists, as the mosthelpful events in therapy sessions (Koerner & Cas-tonguay, 2014). From a clinical standpoint, however,what may be the most important “process” benefit ofPOR is that it can lead to changes in practice. Westet al. (2014) have gathered both empirical andanecdotal evidence indicating that the participationin PRN studies led clinicians to not only modify theirclinical practice but also disseminate the use ofresearch findings and procedures.

Professional Development

While motivated by the goal of improving theoutcome and process of therapy, POR partnersthemselves gain from their collaboration. At onebasic but important level, such partnership allows forthe establishment and growth of connections withothers—locally, across different parts of a country, oraround the world. Both clinicians and researchersdescribed their exchanges with other stakeholders asstimulating and gratifying, as well as supportiveand validating (Adelman et al., 2014; Fernández-Álvarez et al., 2014; Garland & Brookman-Frazee,2014; Koerner & Castonguay, 2014). As described inGarland&Brookman-Frazee (2014), these exchangescan foster reciprocal learning, as with researchersgaining “greater respect for the immediate and oftenrisky clinical challenges therapists faced” and clin-icians having “greater appreciation for the rigor of theresearch process and the ultimate aim of improvingcare” (p.9).

Also related to professional development, theparticipation in POR can provide beneficial trainingexperiences, such as learning strategies to improvethe therapeutic relationship and work with particulartypes of clients (e.g., highly resistant), acquisition ofskills prescribed by specific orientations (cognitive-behavioral, psychodynamic, humanistic, and sys-temic), and increase in awareness of one’s ownpersonal style and its impact on clients (Adelmanet al., 2014; Castonguay et al., 2014; Fernández-Álvarez et al., 2014; Koerner & Castonguay, 2014;Szapocznik et al., 2014; West et al., 2014). Interest-ingly, such learning opportunities are not onlyhelpful to trainees but also to experienced clinicians;

178 L. G. Castonguay et al.

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

as Fernández-Álvarez et al. (2014) learned from theirresearch and practice experience, “[t]eaching andtraining are the most powerful tools for remainingupdated, because they demand contact with newdevelopments and improving training methodologyin the communication of knowledge.” (p. 2)

Various marks of professional recognition can alsoresult from therapists’ (as well as researchers andstudents) engagement in POR, such as publications,conference presentations, research awards, andrequests for consultation (Adelman et al., 2014; Koer-ner & Castonguay, 2014; Szapocznik et al., 2014). Inaddition to providing a source of quality control,outcome monitoring can also be used by clinicians toincrease reimbursement (Koons, O’Rouke, Carter, &Erhardt, 2013). A less public but perhaps moreimportant form of recognition, some therapists havereported having gained credibility in clients’ eyes bytheir association with scientific projects (Koerner &Castonguay, 2014). Another intangible but, nonethe-less, important aspect of professional developmentreported by therapists through a number of PORpartnerships is the sense of purpose and pride gainedfrom contributing to the advancement of science andpractice (e.g., Castonguay et al., 2014; McAleaveyet al., 2014; West et al., 2014). Interestingly, similarfeelings have been reported by clients when agreeing toparticipate in research conducted by their therapists(Castonguay, Nelson et al., 2010).

Organizational Gains

POR benefits are not restricted to individuals,whether these are clients or therapists. Organizationscan also make gains in terms of recognition, qualityof care, and climate. For example, studies based onoutcome monitoring can provide evidence of effec-tiveness, which can be used by administrators andclinicians to increase clients’ positive expectations,referrals from other professionals, and credibility inthe eyes of funding agencies (Adelman et al., 2014;Holmqvist et al., 2014; Szapocznik et al., 2014).Within a particular center or service, collection ofdata can also be used to better understand the needsof clients, as well as to guide the refinement ofinterventions to better address these needs (Adelmanet al., 2014; Holmqvist et al., 2014; McAleavey et al.,2014). As described in Adelman et al. (2014), forinstance, the initial use of outcome monitoring in aresidential center for adolescents with substance-useproblems revealed high levels of violence, bothbefore and at the end of treatment. These unexpec-ted findings led members of the administration and apsychologist to organize the training of the entireclinical staff in a treatment approach specificallytargeting anger. Continued outcome monitoring

showed gradual decrease of anger at posttreatmentduring the training period, as well as the mainten-ance of this improvement after training.

With the same goal of improving the quality of care,large POR partnerships can also provide means tocompare data across sites. For example, in the largePRN infrastructure of university counseling centersdescribed in McAleavey et al. (2014), each sitereceives benchmarked reports allowing administra-tors to contrast the pre- and posttreatment scores ofthe clients they serve with others centers. Both goodand bad news revealed by such reports can providelobbying tools for additional funding and/or policychanges at higher levels of university administration.

In the same way that it can foster interpersonalrelationships among individuals that work in differentworlds (e.g., private practice and university), PORcan also have a positive impact on the culture andclimate within an organization (Castonguay et al.,2014; Garland & Brookman-Frazee, 2014). As a casein point, the success that students in a PRN trainingclinic have had in recruiting their colleagues for theirmasters or doctoral thesis has both relied on andimproved the collaborative attitude that is predomin-ant in many doctoral training programs; an attitudethat could be expressed by many statements, includ-ing, “Graduate school is hell, but we are in togetherand we should do what we can to help friends get theirdegree” (a much more eloquent and well-knownstatement would be “Un pour tous, tous pour un”!Castonguay et al., 2014, p. 9).

Contributing to Health Care System

POR can, and optimally should, also have an impactat a more global level of mental health services, byproviding information about current needs and inter-ventions, as well as by pointing out directions forimprovement. For example, outcome data collectedwithin the context of health-care management havebeen able to predict psychiatric and substance-abusehospitalizations (Boswell et al., 2014). Consideringthe costs (for the individuals, their family, and thesociety) of inpatient treatments, it could be beneficialto use this kind of data to provide targeted, immedi-ate, and more efficient care to those who need themthe most. Interestingly, findings obtained in PRNstudies have already contributed to importantchanges at the national level, such as the increase ofaccess and continuity of psychiatric treatment ingovernmental health programs, and a new policy forassessment and treatment of posttraumatic stressdisorder in the US Army (West et al., 2014). Needlessto say, PRN could be conducted to study the effec-tiveness of those social and clinical interventions.

Psychotherapy Research 179

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

POR findings on the effectiveness and process ofpsychotherapy in general are also available to informpolicy-makers and third-party payers in decisionsregarding implementation and reimbursement ofmental health services. In the current context ofevidence-based practice, the empirical support forthe effectiveness of psychotherapy in naturalisticsettings (see Castonguay et al., 2013) should give itcredence as a high priority form of intervention(Barber, 2009). This recommendation should notbe restricted to problems like depression or anxiety.For example, results from a PRN study show thatthe use of psychotherapy as an evidence-basedrecommendation has yet to be adequately imple-mented in the day-to-day treatment of schizophrenia(West et al., 2014). Other large POR studies havereported similar levels of effectiveness between dif-ferent forms of psychotherapy, including cognitive-behavioral, psychodynamic, and humanistic (seeCastonguay et al., 2013). Such data should dissuadedecision-makers from emphasizing a limited reper-toire of interventions in routine clinical practice(Barber, 2009; Stiles, Barkham, Mellor-Clark, &Connell, 2008). POR findings can also be helpfulto assess the actual use of evidence-based interven-tions in naturalistic settings, as well as to providedirections about how to increase their dissemination,which is one obvious way to facilitate the integrationof science and practice in routine care (Garland &Brookman-Frazee, 2014; Koerner & Castonguay,2014).

Advancing Science

Not only do clinicians (and clients) feel, as wementioned before, that they are contributing to theadvancement of science when they participate inPOR, they actually do. Some features of this type ofresearch are rarely found in studies conducted incontrolled settings; most noteworthy is the access toextremely large samples of therapists (of varioustheoretical orientations), clients (with wide range ofclinical problems), and varying lengths of therapy.With the use of sophisticated statistical analyses thattake into account the nested structure of psychother-apy data, these features offer unique conditions (interms of statistical power and score variance) toinvestigate participant and treatment characteristics,as well as process and outcome variables (e.g., Barber,2009). Because of these distinctive features, and sinceit has been guided in part by clinicians’ interests,POR has led to the much-needed knowledge aboutunderinvestigated treatments (other than cognitive-behavioral), service effects, long-term impact, andcost–benefits of therapy, differential effectiveness oftherapists, and training (Castonguay et al., 2014;

Holmqvist et al., 2014; Strauss et al., 2014)— just toname a few of the innovative contributions.

At its most general level, POR can provide twomajor contributions to the advancement of science.First, because of its particular foci, it can complementmore traditional forms of research (e.g., randomizedstudies in controlled settings) and thus broaden theknowledge base in psychotherapy (Barber, 2009;Barkham & Margison, 2007; Barkham et al., 2010).Second, because some of its findings (with regard tothe alliance, for example) are convergent with thoseobtained in academic settings, POR can increase thestrength of this knowledge. As argued elsewhere,when similar effects are cross-validated across differ-ent methodologies, each with its own strengths andlimitations, we can feel more confident about theveracity and generalizability of these effects (Caston-guay, 2013).

In addition to these general contributions to thefield, POR can also bring local benefits—benefitsthat have more to do with the process of science thanthe content of scientific knowledge. Members ofdifferent clinicians–researchers partnerships havereported that their experiences have generated newand better research. Garland and Brookman-Frazee(2014)’s first PRN with disruptive children, forexample, has served as the basis for later partner-ships on autism. For clinicians in another PRN, oneprimary benefit of conducting research is learninghow to do so (Koerner & Castonguay, 2014).

General Recommendations

The contributors of the present series of papers havealso delineated general recommendations to facilitatethe collaboration of practitioners and researchers inthe conduct and use of research in clinical practice. Anumber of these have already been integrated in theprevious section on fostering strategies. Following area few others, with some of them, as we will highlight,reflecting overarching guidelines that were previouslyoffered for the future of POR.

Technological Advances

To begin with, technological advances should berelied upon. Electronic health records software, forexample, has been found helpful, or at least promis-ing, in the collection of long-term clinical data(McAleavey et al., 2014; West et al., 2014). Elec-tronic technologies can and should be made availableby researchers and administrators to provide clin-icians with easy and immediate tracking and reportingof outcome monitoring (Boswell et al., 2014; Strausset al., 2014). Yet, not all aspects of research shouldmandate the use of sophisticated technology. Boswell

180 L. G. Castonguay et al.

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

et al. (2014), for instance, suggest that alternativemethods of data collection (including paper andpencil options) should be available to therapistsdepending on their preferences.

Instruments Development

In addition to technological developments, instru-ments related to outcome monitoring could berefined and expanded upon. Boswell et al. (2014)have recommended that feedback based on client’sprogress should be benchmarked (and if possible riskadjusted) to identify therapists’ strengths and limita-tions across caseloads or with respect to particulartypes of clients. They also advise that the samefeedback be complemented with clinical supporttools that can provide guidelines for therapists onhow to address difficulties in clients’ lives and/ortreatment that could interfere with change. Holmq-vist et al. (2014) also suggest that service deliverysystems and governmental policies provide access toa set of measures from which therapists could chooseparticular instruments that are best suited to theneeds of individual clients. Such a measurementsystem, they argued, would operate “at a holisticlevel akin to the practitioner working with the wholeperson of the patient rather than with fragmentedparts determined by diagnosis and dominated bysymptom specific measures” (p. 8).

“Just do it!”

While the recommendations above highlight whatresearchers (as well as administrators and policy-makers) can do to facilitate therapists’ engagementin the collection and use of data, several papers in thepresent series clearly demonstrated that therapists donot have to rely on academic researchers to build theirown research programs (Adelman et al., 2014; Fer-nández-Álvarez et al., 2014; Koerner & Castonguay,2014). For therapists who have the time and energy tocombine clinical and empirical work in their ownpractice, our suggestion is simple: “Just do it!”Although the complementarity of expertise andresources can be of great value, partnership withacademicians is not always feasible—it can unfortu-nately be difficult and frustrating (see Adelman et al.,2014). Full-time clinicians should thus be aware ofboth benefits and costs that come along with partner-ing with people living in the world of academia, anddecide for what projects, under what conditions, andto what extent they want or need to collaborate withthem.Moreover, past experiences suggest that POR islikely to be successful if it allows flexibility at the levelof therapist participation. Time, interest, and expert-ise of each clinician should dictate whether he/she

wants to be involved in only one, some, or all aspectsof a study, from the selection of the idea to beinvestigated, the design and implementation of theprotocol, and/or the dissemination of the findings(Koerner & Castonguay, 2014).

Graduate Students

For both the short- and long-term viability of POR, ithas also been recommended to gather the help ofgraduate students. Among the many contributionsthey can offer, students have resources that mostprofessionals, clinicians, and academicians, are shortof: Up-to-date knowledge of methodological andstatistical advances and, most precious of all, timeand energy (Adelman et al., 2014). For students,POR can provide unique opportunities to be involvedin projects (including publications and scientificpresentations) that combine clinical relevance withscientific rigor at an early phase of their career. Theirparticipation might thus be an optimal way to achieveone of several overarching recommendations thatwere recently offered for the future of POR: Beginearly (Castonguay et al., 2013). As stated elsewhere,“simultaneous, seamless, and repeated integration ofscience and practice activities as early as possible in apsychotherapist’s career might create an intellectualand emotional (hopefully secure) attachment toprinciples and merits of the Boulder model” (Caston-guay, 2011, p. 135). A research partnership thatinvolves students, clinicians, and academicians canalso lead to the creation of a pipeline for both archivaland prospective data that cohorts of trainees within auniversity program could have access to (Adelmanet al., 2014). Such pipeline, needless to say, can havelong-term benefits for all stakeholders involved, letalone the field of mental health. To actualize thisbeneficial collaboration, universities should perhapsaccept a sense of responsibility toward preparingtrainees to collect and use data from clinical routine.As argued by Boswell et al. (2014):

training programs should instill the value of collectingroutine data, on both process and outcome, and usingthis information to inform case conceptualization andtreatment planning (Castonguay, Boswell, Constan-tino, Goldfried, & Hill, 2010). In addition, trainingfaculty would do well to encourage an openness toreceiving progress feedback (Boswell & Castonguay,2007), as well as encourage the use of outcomes datato answer clinically relevant research questions earlyon in training. (p. 11)

Networks of Networks

Seeking the engagement of students is one form ofexpansion that has been recommended for the growth

Psychotherapy Research 181

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

of some partnerships. Another one is the creation ofnetworks of networks (Castonguay, 2011). Irrespect-ive of the clinical setting, small research-practiceinfrastructures will be confronted with limitationsin terms of expertise, knowledge, and resources. Inparticular, small partnerships can be restricted intheir ability to recruit large samples of clients andtherapists (e.g., Adelman et al., 2014; Koerner &Castonguay, 2014), which can slow down the collec-tion of required data and/or raise serious concernswith regard to the generalizability of research findings.One recommendation that has been offered toaddress these important issues is for members ofspecific partnerships to connect with other similargroups of partners (Castonguay et al., 2014; Koerner& Castonguay, 2014). The idea is for multiple groupsof clinicians and researchers to work together in thedevelopment of a study, or for one group of partnersto design an investigation and then invite therapistsfrom other networks to join their project by imple-menting the protocol in their own setting. As yetanother overarching recommendation for futurePRNs, this has been referred elsewhere as: “worklocally and collaborate globally” (Castonguay, 2011;Castonguay et al., 2013).

This network of networks is in line with Borkovec’s(2002) dream of a large infrastructure of psychologytraining clinics, all of them linked by a shared basicassessment protocol. Interestingly, this infrastructurecould provide an avenue to address a problem of“extinction” frequently observed in graduate research.As noted by Castonguay et al. (2014), many studentscomplete graduate school without having the time andresources to fully pursue the creative research programthat they began with their masters and/or dissertations.Referring to this problem as the “dusty piles in the lab”effect, they suggest that a large network of trainingclinics could serve as a forum of knowledge exchangeand long-term collaboration, during and after gradu-ate school.

Worldwide Collaboration

The concept of global collaboration can also beapplied to connections among solo practitioners.Koerner and Castonguay (2014) have described atraining initiative where clinicians from around theworld are provided with expert feedback about single-case experiments to test a wide range of hypothesesand interventions in their own clinical practice. Asthey noted, “[t]his line of research begins to build anetwork of therapists and a library of open enrollmentresearch designs and protocols that make it feasible toscale single-case designs to make meaningful con-tributions to the scientific literature” (p. 4). Inaddition to offering a perfect example of local action

and global collaboration, this training infrastructurealso has the potential of fostering three other over-arching recommendations for future POR (Caston-guay et al., 2013). First, to be most valuable andsustainable, this type of research should addressclinicians’ concerns and should be designed, at leastin part, based on their observations and expertise.Nothing comes closer to this recommendation thanhelping practitioners test interventions that they areimplementing, or want to implement with their ownclients. Second, it should add minimally to, or be asconfounded as possible with clinical work. In thiscase, clinicians are simultaneously applying, learning,or refining both therapeutic and empirical skills,thereby reflecting not only a seamless clinical andresearch integration, but an epistemological one.Third, POR has to count. Findings obtained fromnaturalistic settings have to be made known toscholars and decision-makers so that the results canbe taken into account in practice, training, andfunding guidelines. The first step in making PORcount is in dissemination, as when researchers andclinicians work together to create an open library ofscientific contributions. One might also say that whenthey do so, they go further than building bridgesbetween science and practice—a metaphor that sug-gests that clinicians and researchers live on differentbanks of a river and maintain connections by import-ing or exporting knowledge that was independentlysecured. Instead, by blending together their expertiseand resources to directly investigate questions emer-ging from clinical routine, they are creating newlandscapes of knowledge and action (Caston-guay, 2013).

Conclusion

Building POR partnerships is for those who dreambig (McAleavey et al., 2014), not only because of theamount of work required but also for the ambitiousgoals they embrace: Fostering rapprochements ofminds, integrating research and clinical work, andimproving our understanding and practice of themental health field. In their respective pursuit ofthese goals, the contributors of the present serieshave shared their experience about the studies theyhave conducted, challenges they have faced, strat-egies they have adopted to tackle these obstacles, andbenefits that they and their collaborators havegained. They have also offered general suggestionsabout future POR.

Additional lessons can be derived from researchprograms in the field of mental health that have notbeen represented in this series, such as the processand outcome studies by Jacqueline Persons (e.g.,Persons, Roberts, Zalecki, & Brechwald, 2006) and

182 L. G. Castonguay et al.

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

David Burns (e.g., Burns & Nolen-Hoeksema, 1992)in specialized cognitive therapy centers, as well thework that has been conducted within the HealthyFamilies America Research Practice Network(Galano & Schellenbach, 2007) on the preventionof child maltreatment (see Castonguay et al., 2013;McMillen, Lenze, Hawley, & Osbourne, 2009).Moreover, much can be learned from partnershipsof practitioners and researchers in a wide range ofhealth care and medical fields, including nursing,primary care, pediatrics, and family medicine. Nev-ertheless, it is hoped that the breadth of contribu-tions and recommendations captured in the papersof this series will provide both encouragement andguidance to clinicians and researchers to conductand use psychotherapy research in clinical routine.

As part of our attempt to integrate core features ofthe diverse partnerships presented in this series, wehave identified a number of characteristics of suc-cessful POR. Optimally, each study or project con-ducted within such paradigm should be aimed atproviding actionable findings, while imposing min-imal level of extra work, negative consequences, anddrastic changes to clinical practice. We would like toend this paper by offering our thoughts about whatwill make the whole enterprise of POR successful. Ina most basic way, research partnerships in natural-istic settings will be judged as worthwhile, at least inour opinion, if they contribute in the reduction of theproblem that we identified in the introduction of thisseries: The fact that research does not significantlyand substantially influence practice (Castonguay &Muran, 2014). The best and most stringent way toachieve this is perhaps for POR to deliver retainablefindings. Some partnership initiatives have reportedthat participation in research has led to changes inpractice (e.g., West et al., 2014). However, moreefforts will be needed to systematically ensure afeedback loop between the generation of findingsand their implementation in the setting where theyhave been obtained (see Castonguay et al., 2014).Studies should be conducted to inform not only thefield in general but also to quickly and meaningfullytransform the clinical practice that has beeninvestigated.

While research findings should, optimally, beretainable, this does not imply that they have tobecome the only source for clinical guidance. Clinicalexperience, theories, supervision, and training work-shops will, as they should, remain crucial sources ofinfluence. For example, as mentioned above, mon-itoring of outcome data should not be viewed as a wayto replace clinical judgment, but instead, it should beused as one of several complementary tools. Similarly,the success of POR as a whole should not rest onclinicians’ continued involvement in empirical

studies—even those who are members of research-ers–practitioners partnerships. While many therapistsdo seek extra training during their career, most ofthese experiences are time limited. And while theseexperiences can allow them to acquire new and usableskills, it is safe to assume that they do not leadtherapists to abandon their traditional ways of practi-cing. The same expectations should be attached toPOR. Ideally, collaborative research should be per-ceived as opportunities that are available to clinicianswho, at different times in their career, may want tobe engaged in and, as in all learning experiences,might lead to some (but by no mean complete)changes of practice. If these experiences also leadthem to be more interested in research and find waysto improve their practice through the use of empiricalliterature then, in our eyes, POR will have fulfilled itspotential.

References

Adelman, R. W., Castonguay, L. G., Kraus, D. R., & Zack, S.(2014). Conducting research and collaborating with researchers:The experience of clinicians in a residential treatment center.Psychotherapy Research. doi:10.1080/10503307.2014.935520

Barber, J. P. (2009). Towards a working through of some coreconflicts in psychotherapy research. Psychotherapy Research,19, 1–12. doi:10.1080/10503300802609680

Barkham, M., & Margison, F. (2007). Practice-based evidence asa complement to evidence-based practice: From dichotomy tochiasmus. In C. Freeman & M. Power (Eds.), Handbook of

evidence-based psychotherapies: A guide for research and practice

(pp. 443–476). Chichester: Wiley.Barkham, M., Stiles, W. B., Lambert, M. J., & Mellor-Clark, J.

(2010). Building a rigorous and relevant knowledge-base forthe psychological therapies. In M. Barkham, G. E. Hardy, & J.Mellor-Clark (Eds.), Developing and delivering practice-based

evidence: A guide for the psychological therapies (pp. 21–61).Chichester: Wiley.

Borkovec, T. D. (2002). Training clinic research and the possib-ility of a national training clinics practice research network. TheBehavior Therapist, 25, 98–103.

Boswell, J. F., &Castonguay, L. G. (2007). Psychotherapy training:Suggestions for core ingredients and future research. Psychother-apy: Theory, Research, Practice, and Training, 44, 378–383.

Boswell, J. F., Kraus, D. R., Miller, S. D., & Lambert, M. J. (2014).Implementing routine outcome monitoring in clinical practice:Benefits, challenges, and solutions. Psychotherapy Research.doi:10.1080/10503307.2013.817696

Burns, D. D., &Nolen-Hoeksema, S. (1992). Therapeutic empathyand recovery from depression in cognitive-behavioral therapy.Journal of Consulting and Clinical Psychology, 60, 441–449.doi:10.1037/0022-006X.60.3.441.

Castonguay, L. G., Boswell, J. F., Constantino, M. J., Goldfried,M. R., & Hill, C. E. (2010). Training implications of harmfuleffects of psychological treatments. American Psychologist, 65(1),34–49. doi:10.1037/a0017330

Castonguay, L., Nelson, D., Boutselis, M., Chiswick, N., Damer,D., Hemmelstein, N., … Borkovec, T. (2010). Clinicians and/or researchers? A qualitative analysis of therapists’ experiencesin a practice research network. Psychotherapy: Theory, Research,Practice, and Training, 47, 345–354.

Castonguay, L. G. (2011). Psychotherapy, psychopathology, re‐search and practice: Pathways of connections and integration.

Psychotherapy Research 183

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014

Psychotherapy Research, 21(2), 125–140. doi:10.1080/10503307.2011.563250

Castonguay, L. G. (2013). Psychotherapy outcome: A problemworth re-revisiting 50 years later. Psychotherapy: Theory,

Research, Practice, and Training, 50, 52–67.Castonguay, L. G., Barkham, M., Lutz, W., & McAleavey, A. A.

(2013). Practice-oriented research: Approaches and applica-tion. In M. J. Lambert (Eds.), Bergin and Garfield’s Handbook of

psychotherapy and behavior change (6th ed., pp. 85–133). NewYork, NY: Wiley.

Castonguay, L. G., & Muran, J. C. (2014). Fostering collabora-tion between researchers and clinicians through buildingpractice-oriented research: An introduction. Psychotherapy

Research.Castonguay, L. G., Pincus, A. L., & McAleavey, A. A. (2014).

Practice research network in a psychology training clinic: Build-ing an infrastructure to foster early attachment to the scientific-practitioner model. Psychotherapy Research. doi:10.1080/10503307.2013.856045

Fernández-Álvarez, H., Gómez, B., & García, F. (2014). Bridgingthe gap between research and practice in a clinical and trainingnetwork: Aigle’s program. Psychotherapy Research. doi:10.1080/10503307.2013.856047

Galano, J., & Schellenbach, C. J. (2007). Healthy familiesAmerica research practice network: A unique partnership tointegrate prevention sciences and practice. Journal of Preventionand Integration in the Community, 34, 39–66.

Garland, A. F., & Brookman-Frazee, L. (2014). Therapists andresearchers: Advancing collaboration. Psychotherapy Research.doi:10.1080/10503307.2013.838655

Goldfried, M. R., Newman, M., Castonguay, L. G., Fuertes. J.N., Magnavita, J. J., Sobell, L. C., & Wolf, A. W. (2014). Onthe dissemination of clinical experiences in using empiricallysupported treatments. Behavior Therapy, 45, 3–6. doi:10.1016/j.beth.2013.09.007

Holmqvist, R., Philips, B., & Barkham, M. (2014). Developingpractice-based evidence: Benefits, challenges, and tensions.Psychotherapy Research. doi:10.1080/10503307.2013.861093

Hume, D. (1739). A treatise of human nature. London: John Noon.Johnson, J. E. (2014). Integrating psychotherapy research with

public health and public policy goals for incarcerated womenand other vulnerable populations. Psychotherapy Research, 24,229–239. doi:10.1080/10503307.2013.838656

Kazdin, A. E. (2008). Evidence-based treatment and practice: Newopportunities to bridge clinical research and practice, enhancethe knowledge base, and improve patient care. American Psycho-

logist, 63, 146–159. doi:10.1037/0003-066X.63.3.146Koerner, K., &Castonguay, L.G. (2014). Practice-oriented research:

What it takes to do collaborative research in private practice.Psychotherapy Research. doi:10.1080/10503307.2014.939119

Koons, C. R., O’Rouke, B., Carter, B., & Erhardt, E. B. (2013).Negotiating for improved reimbursement for dialectical beha-vior therapy: A successful project. Cognitive and Behavioral

Practice, 20, 314–324. doi:10.1016/j.cbpra.2013.01.003

Kraus, D. R., Castonguay, L. G., Boswell, J. F., Nordberg, S. S.,& Hayes, J. A. (2011). Therapist effectiveness: Implications foraccountability and patient care. Psychotherapy Research, 21,267–276. doi:10.1080/10503307.2011.563249

Krug, S. (2006). Don’t make me think! A common sense approach to

web usability (2nd ed.). Berkeley, CA: New Riders.McAleavey, A. A., Lockard, A. J., Castonguay, L. G., Hayes, J.

A., & Locke, B. D. (2014). Building a practice researchnetwork: Obstacles faced and lessons learned at the Centerfor Collegiate Mental Health. Psychotherapy Research.

McMillen, J. C., Lenze, S. L., Hawley, K. M., & Osbourne, V. A.(2009). Revisiting practice based research networks as aplatform for mental health services research. Administration

and Policy in Mental Health, 36, 308–321. doi:10.1007/s10488-009-0222-2

Lambert, M. J. 2010. Prevention of treatment failure: The use of

measuring, monitoring, and feedback in clinical practice. Washing-ton, DC: American Psychological Association.

Lampropoulos, G. K., Goldfried, M. R., Castonguay, L. G.,Lambert, M. J., Stiles, W. B., & Nestoros, J. N. (2002). Whatkind of research can we realistically expect from the practitioner?Journal of Clinical Psychology, 58, 1241–1264. doi:10.1002/jclp.10109

Persons, J. B., Roberts, N. A., Zalecki, C. A., & Brechwald, W. A.G. (2006). Naturalistic outcome of case-formulation-drivencognitive behavioral therapy for anxiety depressed outpatients.Behaviour Research and Therapy, 44, 1041–1051. doi:10.1016/j.brat.2005.08.005

Stiles, W. B., Barkham, M., Mellor-Clark, J., & Connell, J.(2008). Effectiveness of cognitive–behavioural, person-centred,and psychodynamic therapies in UK primary care routinepractice: Replication in a larger sample. Psychological Medicine,38, 677–688. doi:10.1017/S0033291707001511

Strauss, B., Lutz, W., Steffanowski, A., Wittmann, W., Böhnke, J.R., Rubel, J., …Kirchmann, H. (2014). Benefits and challengesin practice-oriented psychotherapy research in Germany: TheTK and the QS-PSY-BAY projects of quality assurance inoutpatient psychotherapy. Psychotherapy Research.

Szapocznik, J., Muir, J. A., Duff, J., & Schwartz, S. J. (2014). Briefstrategic family therapy: Implementing evidence-based modelsin community settings. Psychotherapy Research.

Youn, S., Kraus, D. R., & Castonguay, L. G. (2012). Treatmentoutcome package: Facilitating practice and clinically relevantresearch. Psychotherapy: Theory, Research, Practice, and Training,49, 115–122.

West, J. C., Mościcki, E. K., Duffy, F. F., Wilk, J. E., Countis, L.,Narrow,W. E., & Regier, D. A. (2014). APIRE practice researchnetwork: Accomplishments, challenges, and lessons learned.Psychotherapy Research.

Zarin, D. A., Pincus, H. A., West, J. C., & McIntyre, J. S. (1997).Practice-based research in psychiatry. American Journal of

Psychiatry, 154, 1199–1208.

184 L. G. Castonguay et al.

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

11:

27 0

4 D

ecem

ber

2014