Phyllis C. Panzano, Ph.D. , PI Dee Roth, M.A., Co-PI Bev Seffrin, Ph.D, Senior Consultant Dushka Crane-Ross, Ph.D., Project Manager Decision Support Services, Inc. Ohio Dept of Mental Health, OPER The Innovation Diffusion and Adoption Research Project (IDARP) ODMH RESEARCH RESULTS BRIEFING 2003 Funded by the ODMH & the Mac Arthur Foundation
117
Embed
Phyllis C. Panzano, Ph.D., PI Dee Roth, M.A., Co-PI Bev Seffrin, Ph.D, Senior Consultant Dushka Crane-Ross, Ph.D., Project Manager Decision Support Services,
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Phyllis C. Panzano, Ph.D. , PI Dee Roth, M.A., Co-PI
4)4) Integrated Dual Disorder Treatment (IDDT) –Integrated Dual Disorder Treatment (IDDT) –New Hampshire - Dartmouth model New Hampshire - Dartmouth model
Research Design
o Longitudinal studyLongitudinal study
o Organizations at different stages of Organizations at different stages of adoptionadoption
o Multiple key informants at each Multiple key informants at each organizationorganization
o Quantitative and qualitative dataQuantitative and qualitative data
o Interviews, surveys & archival dataInterviews, surveys & archival data
Participating Projects*by Type of Innovation
23
37
16 15
0
10
20
30
40
Fre
quen
cy
MST
OM
AP
IDD
T/SA
MI
Clu
ster
Alli
anc
e
*18 organizations involved in multiple projects; Total of 74 organizations with 91 projects under study.
Participating Projectsby Stage of Adoption at Time One
12 179
47
605
101520253035404550
Fre
quen
cy
Impl
emen
ter
De-
adop
ter
Ado
pter
Nev
erW
ait &
see
N = 91
Participating Projectsby Stage of Adoption at Time Two
1 1 2
42
405
101520253035404550
Fre
quen
cy
Impl
emen
ter
De-
adop
ter
Ado
pter
Nev
erW
ait &
See
N = 50
Key Informants by Level at Time One
45
10781
6175
0
25
50
75
100
125
Fre
quen
cy
CFO
/QA
Impl
emen
ter
Dec
isio
n m
aker C
CO
E
Com
mun
ity
Col
labo
rative
N = 369
Key Informants by Level at Time Two
1139
64
210
25
50
75
100
125
Fre
quen
cy
Impl
emen
ter
Dec
isio
n m
aker
CCO
E
Com
mun
ity
Col
labo
rative
N = 135
Findings
622
agree
Strongly disagree
Very satisfied
$ 37,500
Do the data support our four models?
THE TIP….OF THE TIP
___________________________________
POSITIVE CORRELATION
As the value of one variable increases,
the value of a second variable also increases
Median Income
Years of Formal Education
Lower
Higher
Less More
+ correlation (r = +1.00)
___________________________________
NEGATIVE CORRELATION
As the value of one variable increases,
the value of a second variable decreases
Unemployment
Rate
Years of Formal Education
Lower
Higher
Less More
- correlation (r = -1.00)
___________________________________
ZERO ‘0’ CORRELATION
The relationship between the value of one variable and the value of a second variable is
random
Years of Formal Education
Shorter
Taller
Less More
Zero Correlation
Height
x
x
x
x
x
x
x
x
x
x
x
x
x
x x
x
x
(r = 0.00)
___________________________________
CORRELATION CANNOT BE DETERMINED
because the value of one (or both) variable(s) is constant or almost constant
Lower
Higher
Years of Formal Education = BA, BS
Unemployment
Rate
The Adoption Decision (Model 1)
Time 1/First contact data
Phase 1: A Decision Under Risk
Likelihood of implementing as
indicated by Stage
Implementer Adopter Wait & See Never
Perceived
Risk of
Adopting
Capacity to
Manage or
Absorb Risk
Risk-taking
Propensity
-.51
+.38
+.20
Antecedents to Risk Perceptions
Likelihood of implementing as
indicated by Stage
Implementer Adopter Wait & See Never
Perceived
Risk of
Adopting
Capacity to
Manage
Risk
Risk-taking
Propensity
-.51
+.38
+.20
ANT E C E D ENT S
Antecedents to Perceived Risk
Perceived
Risk
Innovation Level Factors
• Relative Advantage
• Scientific Evidence
• Experiential Evidence
Org-Level Factors
• Knowledge Set
-.20
-.30
-. 43
-.51
Environmental Factors
• Norms for Adoption
-.45
Antecedents to Risk Management
Capacity to
Manage Risk
EBP–Level Factors
Ease of Use
Org–Level Factors
Top Mgmt. Support
Environmental Factors
Environmental uncertainty
+ .45
+ .50
- .22
Craft Skills+ .2
5
Dedicated Resources
+ .63
Antecedents to Risk Propensity
Organization-Level Factors
Learning Encouragement
Managerial Attitude
About Change
Risk
Propensity
+.71
+.23
Summing Up: Model 1
3. Antecedents have implications for action3. Antecedents have implications for action
1. Adoption decision is a decision 1. Adoption decision is a decision involving riskinvolving risk2. Organizations are more likely to adopt 2. Organizations are more likely to adopt if:if:
Perceived risk of adopting is lowPerceived risk of adopting is low
Capacity to manage risk is highCapacity to manage risk is high
Propensity to take risks is highPropensity to take risks is high
Model 2: Implementation Phase
Understanding Outcomes of Understanding Outcomes of Implementation Implementation
for Adopters and Implementersfor Adopters and Implementers
Time 2/Second Contact Data
Two classes of outcomes
1. Implementation Effectiveness
(e.g., fidelity, assimilation)
2. Innovation/practice Effectiveness
(e.g., positive outcomes)
Is implementation effectiveness related to innovation effectiveness?
IMPLEMENTATION EFFECTIVENESS
INNOVATION EFFECTIVENESS
Reinvention1
Positive outcomes
?
-.64
Assimilation Positive outcomes
.61
1 Self report; reflects extent to which practice was modified
Level 4: Inter-organizational
Level 3: Adopting organization
Level 2: Project level
Level 1: Innovation level
Dependent Variables:
• Implementation effectiveness
• Innovation effectiveness
Level 5: Environmental Model 2
Assimilation: One measure of
implementation effectiveness
Is Assimilation Explained by Variables at Multiple Levels?
DYAD: Communication quality +.45
ORG: Learning culture +.30
Centralization +.43
PROJECT: Dedicated resources +.52
Ease of use +.40
INNOV: Fit w/Tx philosophy +.45
Dependent Variable: Assimilation*
* Extent practice seen as part of permanent operations
Some Examples
Variables at multiple levels are related to reported
assimilation.
Are views about positive outcomes explained by
variables at multiple levels?
Positive outcomes
Overall positive consequencesOverall positive consequences Positive outcomes for consumersPositive outcomes for consumers Positive impact on organization’s imagePositive impact on organization’s image Positive impact on organization Positive impact on organization
Interview team produces transcripts from Interview team produces transcripts from interviews with multiple agency sourcesinterviews with multiple agency sources
Qualitative Qualitative “codes”“codes” attached to text in Atlas Ti attached to text in Atlas Ti Software program such as:Software program such as:
Issue diagnosis & decision processIssue diagnosis & decision process Planning process for EBP implementationPlanning process for EBP implementation Facilitators & BarriersFacilitators & Barriers Expected/unexpected, +/- outcomesExpected/unexpected, +/- outcomes
Unit of Analysis =Unit of Analysis = “mentions” “mentions” or coded phrasesor coded phrases
Focus of today’s look at qualitative data
Data collected during time one/first contact Data collected during time one/first contact with 36 projects (~ 3 interviews per)with 36 projects (~ 3 interviews per)
Projects X EBPProjects X EBP 13 IDDT13 IDDT 9 Cluster-Based Planning9 Cluster-Based Planning 7 OMAP7 OMAP 7 MST7 MST
Glossary of Categories
CCOECCOE - relating to the CCOE, its staff and - relating to the CCOE, its staff and services it provides.services it provides.
EBPEBP – perceptions relating to the innovation. – perceptions relating to the innovation.
MoneyMoney - expenses (actual or anticipated), - expenses (actual or anticipated), funding of the EBP and financial issues that funding of the EBP and financial issues that impact the agency.impact the agency.
Staff Staff - reactions, recruitment, retention and - reactions, recruitment, retention and qualifications of staff. qualifications of staff.
System System - coordination, collaboration, and - coordination, collaboration, and interest in Mental Health and other related interest in Mental Health and other related systems.systems.
CCOE: Major Themes
Facilitators (n = 119)Facilitators (n = 119)
Attended CCOE Attended CCOE presentation/became presentation/became aware of CCOE/had aware of CCOE/had previous experience previous experience with (n = 55)with (n = 55)
CCOE provides CCOE provides instrumental help instrumental help (n = 50)(n = 50)
Positive reaction to Positive reaction to CCOE (n = 11)CCOE (n = 11)
Barriers (n = 36)Barriers (n = 36)
CCOE doesn’t CCOE doesn’t understand the understand the agency’s issues or agency’s issues or constraints (n = 9)constraints (n = 9)
CCOE: Mentions by Stage
20
51
11
37
5
27
220
25
50
75
100
125
150
Wait & See/
Never
Adopter Implementer Deadopter
Facilitators Barriers
EBP: Major Themes
Facilitators (n = 225)Facilitators (n = 225)
Received training or Received training or information (n = 53)information (n = 53)
EBP is good match with EBP is good match with culture, systems, or culture, systems, or similar to what we similar to what we already do (n = 24)already do (n = 24)
EBP might be useful EBP might be useful (n = 19)(n = 19)
Barriers (n = 153)Barriers (n = 153)
Don’t know how to Don’t know how to proceed – in the dark proceed – in the dark (n = 24)(n = 24)
EBP isn’t a good fit to EBP isn’t a good fit to this organization this organization (n = 13)(n = 13)
EBP might NOT be EBP might NOT be useful (n = 13)useful (n = 13)
There is a potential There is a potential savings from the EBP savings from the EBP (not necessarily for the (not necessarily for the agency) (n = 6)agency) (n = 6)
Barriers (n = 115)Barriers (n = 115)
Agency has financial Agency has financial issues/EBP costs issues/EBP costs money (n = 80)money (n = 80)
Funding for the EBP is Funding for the EBP is not sustainable (n = not sustainable (n = 29)29)
Resistance to EBP, Resistance to EBP, skepticism, lack of skepticism, lack of interest (n = 44)interest (n = 44)
Competing priorities Competing priorities (e.g. innovation vs. (e.g. innovation vs. productivity) (n = 15)productivity) (n = 15)
Staff: Mentions by Stage
82
5 10
25 24
112
1619
0
25
50
75
100
125
150
Wait & See/
Never
Adopter Implementer Deadopter
Facilitators Barriers
System: Major Themes
Facilitators (n = 106)Facilitators (n = 106)
Support and interest in Support and interest in the system (n = 34)the system (n = 34)
Collaboration/ Collaboration/ cooperation/ cooperation/ communication & communication & integration in the integration in the system (n = 33)system (n = 33)
Barriers (n = 82)Barriers (n = 82)
Lack of support/no Lack of support/no interest (n = 29)interest (n = 29)
Conflict, lack of Conflict, lack of collaboration between collaboration between important entities, no important entities, no communication (n = 29)communication (n = 29)
Competing priorities & Competing priorities & turmoil in system (n = turmoil in system (n = 9)9)
Overall, facilitators Overall, facilitators were mentioned were mentioned more often than more often than barriers (641:557).barriers (641:557).
EBP: The category EBP: The category with the most with the most mentions of mentions of facilitators (225);facilitators (225);
Staff: The category Staff: The category with the most with the most mentions of mentions of barriers (171).barriers (171).
CCOE
EBP
$
Staff
System
CCOE
EBP
$
StaffSystem
Facilitator & Barrier Analysis -Phase
Facilitators and barriers can usually be Facilitators and barriers can usually be identified as occurring during specific identified as occurring during specific phasesphases of the process. of the process.
The next analysis separates most of the The next analysis separates most of the same “mentions” in terms of phase in same “mentions” in terms of phase in which they occurred.which they occurred.
Facilitator & Barrier Analysis – Initiation Phase
Initiation PhaseInitiation Phase – A facilitator or barrier – A facilitator or barrier that is anticipated or experienced PRIOR that is anticipated or experienced PRIOR to the adoption decision. to the adoption decision.
Early ImplementationEarly Implementation – A facilitator or – A facilitator or barrier that is anticipated or barrier that is anticipated or experienced AFTER the adoption experienced AFTER the adoption decision, but before full implementation. decision, but before full implementation.
Early ImplementationEarly Implementation Facilitators = 166 Facilitators = 166 Early Implementation Early Implementation Barriers = 122Barriers = 122
Mentions during Early Implementation
103
011
52
15
76
31
00
25
50
75
100
125
150
175
Wait & See/
Never
Adopter Implementer Deadopter
Facilitators Barriers
Implementation Phase
ImplementationImplementation – A facilitator or barrier – A facilitator or barrier that is anticipated or experienced that is anticipated or experienced AFTER the agency begins to implement AFTER the agency begins to implement the EBP.the EBP.
Initiation:Initiation: Facilitators are mentioned more Facilitators are mentioned more than TWICE AS FREQUENTLY as barriers.than TWICE AS FREQUENTLY as barriers.
Early Implementation:Early Implementation: Adopters & Adopters & Implementers mention 50% more facilitators Implementers mention 50% more facilitators than barriers (trend not seen in Wait & than barriers (trend not seen in Wait & See/Never or Deadopters).See/Never or Deadopters).
Implementation:Implementation: While there are more While there are more barriers than facilitators mentioned throughout barriers than facilitators mentioned throughout the Implementation phase, Deadopters the Implementation phase, Deadopters mention nearly FOUR TIMES more barriers mention nearly FOUR TIMES more barriers than facilitators.than facilitators.
So what does all this mean?
Major Messages
Adoption decision is a risky decision Adoption decision is a risky decision
Implementation effectiveness related to but Implementation effectiveness related to but not equal to innovation effectivenessnot equal to innovation effectiveness
Factors at many levels contribute to successFactors at many levels contribute to success
What happens early (e.g., during initiation) can What happens early (e.g., during initiation) can have enduring effects have enduring effects