19th Annual RTC Conference Presented in Tampa, February 2006 1 CASRC Multiple Stakeholder Perspectives Multiple Stakeholder Perspectives on Evidence-Based Practice on Evidence-Based Practice Implementation Implementation Gregory A. Aarons, Ph.D. Gregory A. Aarons, Ph.D. 1,2,3 1,2,3 Karen Karen Zagursky Zagursky, B.A. , B.A. 1,3 1,3 Larry Palinkas, Ph.D. Larry Palinkas, Ph.D. 1,4 1,4 1 Child and Adolescent Services Research Center (CASRC) Child and Adolescent Services Research Center (CASRC) 1 Children Children’s Hospital San Diego, CA s Hospital San Diego, CA 2 San Diego State University San Diego State University 3 University of California, San Diego University of California, San Diego 4 University of Southern California University of Southern California 1, 1, 3020 Children’s Way, MC 5033 San Diego, CA 92123 [email protected](858) 966-7703 ext. 3550 http://casrc.org CASRC Acknowledgements Acknowledgements NIMH NIMH – R03 R03 MH070703 MH070703 (Aarons) Concept (Aarons) Concept Mapping of Readiness for Evidence-Based Mapping of Readiness for Evidence-Based Practice Practice NIMH - R01 MH072961 (Aarons) Mixed- NIMH - R01 MH072961 (Aarons) Mixed- Methods Study of a Statewide EBP Methods Study of a Statewide EBP Implementation Implementation NIMH P30 MH068579 (Proctor) Pilot Study: NIMH P30 MH068579 (Proctor) Pilot Study: Aarons PI Organizational Receptivity to Aarons PI Organizational Receptivity to Evidence-Based Practice Evidence-Based Practice CASRC Agenda Agenda The need for effective implementation The need for effective implementation Barriers and Facilitators to Implementing EBP Barriers and Facilitators to Implementing EBP Study methods Study methods Results Results What does it all mean? What does it all mean? CASRC EBP Implementation is Happening EBP Implementation is Happening Effective implementation of EBPs into real-world service Effective implementation of EBPs into real-world service settings is important for improving service quality and settings is important for improving service quality and outcomes for youth outcomes for youth (Hoagwood & Olin, 2002; Jensen, 2003) (Hoagwood & Olin, 2002; Jensen, 2003) Some (but not many) implementation improvement Some (but not many) implementation improvement methods are being tried methods are being tried (Haynes & Haines, 1998) (Haynes & Haines, 1998) – Abstracting services Abstracting services – Evidence-based clinical guidelines Evidence-based clinical guidelines – Incentives for better care systems Incentives for better care systems – Increasing effectiveness of quality improvement programs Increasing effectiveness of quality improvement programs Research is testing some factors associated with Research is testing some factors associated with implementation but multiple stakeholder perspectives are implementation but multiple stakeholder perspectives are not well defined not well defined (NIMH R01, R03, PI: Aarons; R01 Webster Stratton, (NIMH R01, R03, PI: Aarons; R01 Webster Stratton, R01 Chaffin, R34 Shipp, ) R01 Chaffin, R34 Shipp, ) CASRC We are Learning about Implementation We are Learning about Implementation Some barriers to implementation have been Some barriers to implementation have been identified identified – e.g., lack of funds for continuing education (Simpson, e.g., lack of funds for continuing education (Simpson, 2002). 2002). We know little about the most effective manner We know little about the most effective manner in which to implement EBPs in which to implement EBPs –(Henggeler Henggeler, Lee, & Burns, 2002; Morgenstern, 2000) , Lee, & Burns, 2002; Morgenstern, 2000) New models of implementation have been New models of implementation have been developed developed – (Aarons, 2005; (Aarons, 2005; Frambach Frambach & & Schillewaert Schillewaert , 2002; Klein, , 2002; Klein, Conn Conn,& ,& Sorra Sorra, 2002). , 2002). CASRC Implementation is Complex Implementation is Complex Implementation should be evidence-based Implementation should be evidence-based Implementation is a multilevel issue (Dixon et al., 2001). Implementation is a multilevel issue (Dixon et al., 2001). – Policies Policies – Agencies Agencies – Programs Programs – Administrative staff Administrative staff – Clinicians Clinicians – Consumers Consumers Clear, comprehensive, measurable, and testable Clear, comprehensive, measurable, and testable implementation models are needed to guide research on implementation models are needed to guide research on organizational change organizational change There are few empirical studies addressing these issues in There are few empirical studies addressing these issues in youth mental health services youth mental health services
6
Embed
19th Annual RTC Conference Presented in Tampa, …rtckids.fmhi.usf.edu/rtcconference/handouts/pdf/19...19th Annual RTC Conference Presented in Tampa, February 2006 1 CASRC Multiple
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
on Evidence-Based Practiceon Evidence-Based Practice
ImplementationImplementation
Gregory A. Aarons, Ph.D.Gregory A. Aarons, Ph.D.1,2,31,2,3
Karen Karen ZagurskyZagursky, B.A. , B.A. 1,31,3
Larry Palinkas, Ph.D. Larry Palinkas, Ph.D. 1,41,4
11Child and Adolescent Services Research Center (CASRC)Child and Adolescent Services Research Center (CASRC)11ChildrenChildren’’s Hospital San Diego, CAs Hospital San Diego, CA
22San Diego State UniversitySan Diego State University33University of California, San DiegoUniversity of California, San Diego
44University of Southern CaliforniaUniversity of Southern California1,1,
NIMH NIMH –– R03 R03 MH070703MH070703 (Aarons) Concept (Aarons) ConceptMapping of Readiness for Evidence-BasedMapping of Readiness for Evidence-BasedPracticePractice
NIMH - R01 MH072961 (Aarons) Mixed-NIMH - R01 MH072961 (Aarons) Mixed-Methods Study of a Statewide EBPMethods Study of a Statewide EBPImplementationImplementation
NIMH P30 MH068579 (Proctor) Pilot Study:NIMH P30 MH068579 (Proctor) Pilot Study:Aarons PI Organizational Receptivity toAarons PI Organizational Receptivity toEvidence-Based PracticeEvidence-Based Practice
CASRC
AgendaAgenda
The need for effective implementationThe need for effective implementation
Barriers and Facilitators to Implementing EBPBarriers and Facilitators to Implementing EBP
Study methodsStudy methods
ResultsResults
What does it all mean?What does it all mean?
CASRC
EBP Implementation is HappeningEBP Implementation is Happening
Effective implementation of EBPs into real-world serviceEffective implementation of EBPs into real-world servicesettings is important for improving service quality andsettings is important for improving service quality andoutcomes for youth outcomes for youth (Hoagwood & Olin, 2002; Jensen, 2003)(Hoagwood & Olin, 2002; Jensen, 2003)
Some (but not many) implementation improvementSome (but not many) implementation improvementmethods are being tried methods are being tried (Haynes & Haines, 1998)(Haynes & Haines, 1998)
–– Incentives for better care systemsIncentives for better care systems
–– Increasing effectiveness of quality improvement programsIncreasing effectiveness of quality improvement programs
Research is testing some factors associated withResearch is testing some factors associated withimplementation but multiple stakeholder perspectives areimplementation but multiple stakeholder perspectives arenot well defined not well defined (NIMH R01, R03, PI: Aarons; R01 Webster Stratton,(NIMH R01, R03, PI: Aarons; R01 Webster Stratton,R01 Chaffin, R34 Shipp, )R01 Chaffin, R34 Shipp, )
CASRC We are Learning about ImplementationWe are Learning about Implementation
Some barriers to implementation have beenSome barriers to implementation have beenidentifiedidentified
–– e.g., lack of funds for continuing education (Simpson,e.g., lack of funds for continuing education (Simpson,2002).2002).
We know little about the most effective mannerWe know little about the most effective mannerin which to implement EBPsin which to implement EBPs
Implementation is ComplexImplementation is Complex
Implementation should be evidence-basedImplementation should be evidence-based
Implementation is a multilevel issue (Dixon et al., 2001).Implementation is a multilevel issue (Dixon et al., 2001).–– PoliciesPolicies
–– AgenciesAgencies
–– ProgramsPrograms
–– Administrative staffAdministrative staff
–– CliniciansClinicians
–– ConsumersConsumers
Clear, comprehensive, measurable, and testableClear, comprehensive, measurable, and testableimplementation models are needed to guide research onimplementation models are needed to guide research onorganizational changeorganizational change
There are few empirical studies addressing these issues inThere are few empirical studies addressing these issues inyouth mental health servicesyouth mental health services
19th Annual RTC Conference
Presented in Tampa, February 2006
2
CASRC
Goals of the StudyGoals of the Study
To identify barriers and facilitators ofTo identify barriers and facilitators of
adoption of EBPs for organizations servingadoption of EBPs for organizations serving
youth with Mental Health disordersyouth with Mental Health disorders
Examine what various stakeholder groupsExamine what various stakeholder groups
identify as most important and mostidentify as most important and most
changeable.changeable.
CASRC
Methods IMethods I
Programs within agencies selected based on:Programs within agencies selected based on:
–– Types of Services ProvidedTypes of Services ProvidedOutpatientOutpatient
Day TreatmentDay Treatment
Case ManagementCase Management
Residential/InpatientResidential/Inpatient
–– Size of AgencySize of AgencyLarge and SmallLarge and Small
–– Size of ProgramSize of ProgramLarge and SmallLarge and Small
–– LocationLocationUrban vs. RuralUrban vs. Rural
CASRC
Participant SelectionParticipant Selection
Selected programs were either operated by theSelected programs were either operated by theCounty or provided contract services to theCounty or provided contract services to thecounty.county.
Organizational structures varied by level ofOrganizational structures varied by level ofbureaucracy and fiscal constraints on servicesbureaucracy and fiscal constraints on services(Aarons, 2004)(Aarons, 2004)
Individual participants selected by snowballIndividual participants selected by snowballsamplingsampling
CASRC
Sample SelectionSample Selection
Participants drawn from 6 organizationalParticipants drawn from 6 organizationallevels:levels:
Policy: County Mental Health Officials (n = 6)Policy: County Mental Health Officials (n = 6)
–– Qualitative methods used to generate dataQualitative methods used to generate data
–– Data analyzed using quantitative methodsData analyzed using quantitative methods
Begin with structured brainstormingBegin with structured brainstorming
–– Participants Participants generategenerate and then and then useuse a focus a focus
statement to guide identifying barriers andstatement to guide identifying barriers and
facilitators to implementationfacilitators to implementation
CASRC
ProcedureProcedure
Focus statementFocus statement
–– ““What are the factors that influence theWhat are the factors that influence the
acceptance and use of evidence-basedacceptance and use of evidence-based
practices in publicly funded mental healthpractices in publicly funded mental health
programs for families and children?programs for families and children?””
Independent stakeholder groupIndependent stakeholder group
brainstormingbrainstorming
Statements combined across all groupsStatements combined across all groups
CASRC
ProcedureProcedure
““UnstructuredUnstructured”” Card Sort Card Sort
–– 105 Statements105 Statements
–– All participants sort the same statementsAll participants sort the same statements
–– Sorted based on similaritySorted based on similarity
–– >1 pile>1 pile
Statement RatingsStatement Ratings
–– "Importance"Importance““
–– "Changeability""Changeability"
–– 0 to 4 point scale0 to 4 point scale(Not at all (Not at all !! A very great extent) A very great extent)
CASRC AnalysisAnalysis
Multidimensional scaling (MDS) and cluster analysisMultidimensional scaling (MDS) and cluster analysis
MDS analysis results in a MDS analysis results in a ““mapmap”” of the conceptual space of the conceptual spacewith similar issues closer togetherwith similar issues closer together
Solution represents psychological Solution represents psychological ““distancedistance”” or similarity or similaritybetween conceptsbetween concepts
Statements more similar in meaning are closer togetherStatements more similar in meaning are closer together
Statements grouped into non-overlapping categoriesStatements grouped into non-overlapping categoriescalled clusterscalled clusters
Clusters closer together are more conceptually relatedClusters closer together are more conceptually related
CASRC
ResultsResults
Fourteen overall clusters best fit dataFourteen overall clusters best fit data
One overall solution for all participantsOne overall solution for all participants
–– Participants reconvene to Participants reconvene to ““make sensemake sense”” of of
solutionsolution
–– Cluster namingCluster naming
Importance ratings overlaid on solutionImportance ratings overlaid on solution
CASRC
14 Clusters14 Clusters
"#"# Clinical PerceptionsClinical Perceptions
$#$# Staff Development & SupportStaff Development & Support
%#%# Staffing ResourcesStaffing Resources
&#&# Agency CompatibilityAgency Compatibility
'#'# EBP LimitationsEBP Limitations
(#(# Consumer ConcernsConsumer Concerns
)#)# Impact on Clinical PracticeImpact on Clinical Practice
*#*# Beneficial Features (of EBP)Beneficial Features (of EBP)
There are a number of multiple stakeholderThere are a number of multiple stakeholder
concerns that may impact implementation ofconcerns that may impact implementation of
EBPs in real world service settings.EBPs in real world service settings.
Groups varied on Importance and ChangeabilityGroups varied on Importance and Changeability
ratings.ratings.
It is important to consider the concerns ofIt is important to consider the concerns of
multiple stakeholders in EBP implementation.multiple stakeholders in EBP implementation.
CASRC
ConclusionsConclusions
Processes for egalitarian multiple stakeholders input canProcesses for egalitarian multiple stakeholders input canfacilitate cultural exchangefacilitate cultural exchange
Stakeholder perspectives can inform implementationStakeholder perspectives can inform implementationprocessprocess
Examples:Examples:
–– Optimizing message content may promote more positiveOptimizing message content may promote more positiveattitudes toward implementation of change in service modelsattitudes toward implementation of change in service models
–– Staff issues need to be addressed up front to promoteStaff issues need to be addressed up front to promoteimplementation effectivenessimplementation effectiveness
Further research is needed to better understand howFurther research is needed to better understand howfactors identified in the present study impact actual EBPfactors identified in the present study impact actual EBPimplementation efforts.implementation efforts.
19th Annual RTC Conference
Presented in Tampa, February 2006
6
CASRC Reference List
Aarons, G. A. (2005). Measuring provider attitudes toward evidence-based practice: Organizational
context and individual differences. Child and Adolescent Psychiatric Clinics of North America, 14,
255-271.
Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice:
The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research, 6(2), 61-
74.
Dixon, L., McFarlane, W. R., Lefley, H., Lucksted, A., Cohen, M., Falloon, I. et al. (2001). Evidence-
based practices for services to families of people with psychiatric disabilities. Psychiatric Services,
52(7), 903-910.
Haynes, B., & Haines, A. (1998). Barriers and bridges to evidence based clinical practice. Biomedical
Journal, 317, 273-276.
Henggeler, S. W., Lee, T., & Burns, J. A. (2002). What happens after the innovation is identified?
Clinical Psychology: Science and Practice, 9(2), 191-194.
Hoagwood, K., & Olin, S. (2002). The NIMH blueprint for change report: Research priorities in child
and adolescent mental health. Journal of the American Academy of Child and Adolescent
Psychiatry, 41(7), 760-767.
Jensen, P. S. (2003). Commentary: The next generation is overdue. Journal of the American Academy
of Child and Adolescent Psychiatry, 42(5), 527-530.
Morgenstern, J. (2000). Effective technology transfer in alcoholism treatment. Substance Use &
Misuse, 35(12-14), 1659-78.
Simpson, D. D. (2002). A conceptual framework for transferring research to practice . Journal of
Substance Abuse Treatment, 22(4), 171-182.
Trochim, W. M., K, Cook, J. A., & Setze, R. J. (1994). Using concept mapping to develop a conceptual
framework of staff's views of a supported employment program for individuals with severe mental