Measurement and analysis of provider behaviour change in knowledge translation research: Is there a best practice?
2nd National Knowledge Translation Conference in Rehabilitation, April 19, 2018
Nancy Salbach, PT, PhDCIHR New Investigator in Knowledge Translation
Associate Professor, Department of Physical Therapy, University of Toronto
Presentation Objectives:
1. To consider the advantages and limitations of methods for measuring provider behaviour change following a KT intervention
2. To understand which quantitative approaches to measuring provider behaviour change may lead to clustered data and how to evaluate clustering in data analysis
2
Outcomes of KT studies in RehabilitationSystematic review: Jones et al 2015
3
More than 75% of KT intervention studies* in rehab target professional or process outcomes
*provider targeted: PT (13 studies); OT (7 studies); SLP (3 studies); OT&PT (3 studies)
Systematic review: Jones et al 2015 (n=26 studies)
4
Why professional or process outcomes?• Ultimate aim = improved health of Canadians =
improved patient function in rehabilitation setting
• KT interventions targeting multidisciplinary team
• Patient function = primary outcome
• Provider practice = intermediate ‘process’ outcome
• KT interventions targeting uniprofessional practices
• Provider practice = primary outcome as patient function influenced by practices of “other” providers
5
Most professional-level outcomes measured by provider self-report
Systematic review: Jones et al 2015
6
Warning! Brain squeeze
What are the pros and cons of different methods for measuring behaviour change in rehabilitation providers following KT interventions?
7
https://imgflip.com/meme/58388563/
Thinking-hard
Provider Self-report
Pros
• Feasible
• Inexpensiveif online
• Data on large group
Cons
• Measurement at one point in time
• Attrition over time
• Social desirability bias
• Item reliability?
8
Patient Self-report
Pros
• Inexpensive if online
• Can capture practice over time
Cons
• Privacy - Difficulty obtaining patient emails
• Recall
• Expense - phone/mail survey
• Low response rate
• Is response linked to health?9
Interviews & Focus Groups
Pros
• Deep understanding of mechanisms of behaviour change
Cons
• Interviews=resource intensive
• Focus groups: scheduling and opportunity to speak
• Limited for large groups10
Observing Provider Practice
Pros
• Observer unbiased
• Range of patients
Cons
• Expensive, resource intensive
• Representativeness depends on duration of observation
• Observation may influence practice11
https://www.smartcherry.in/your-
observation-power-smart-cherry/
Health record review
Pros
• Retrospective: Less vulnerable to bias
• Range of patients
Cons
• Inconsistent documentation
• Expensive, resource intensive
• If prospective, providers can change documentation behaviour
12
http://www.bcm
j.org/mds-
be/merit-and-
agony-
retrospective-
chart-reviews-
medical-
student%E2%8
0%99s-
perspective
Now a word about clustering
• Clustering is when people or data in your study form natural groupings because the people or data within a group are more related to each other than they are to people or data outside the group
• Can you think of examples of clustering in rehabilitation?
13
Clustering of Providers within HospitalsEg. Guideline implementation study, examine provider use of recommendations in 3 rehab hospitals
14
Hospital 1 Hospital 2 Hospital 3
Clusters
Clustering of Patients within ProvidersEg. Guideline implementation study, examine provider use of recommendations with each patient
15
Provider 1 Provider 2
Patients Patients
Provider 3
Patients
Clusters
Clustering of Measurements within PatientsEg. Guideline implementation study, examine provider use of recommendations across sessions with same patients
16
Patient 1 Patient 2
Data points Data points
Patient 3
Data points
Clusters
Multi-level Clustering
Data
points
Hospital
Hospital 2 Hospital 3Cluster Level
Patients
Provider
What happens when you don’t account for clustering?
• It depends on whether clustering has an influence
• If clustering influences your study outcome, and you don’t account for clustering in the analysis, you will likely get the wrong answer
• Let’s take a look at which level of clustering was important in a guideline implementation trial
18
Stroke Canada Optimization of Rehabilitation through Evidence-Implementation Trial (SCORE-IT)
• National cluster randomized guideline implementation trial
• 20 stroke inpatient rehabilitation centres randomized to a multi-component facilitated or passive KT intervention for guideline implementation
• Primary outcome was patient function: Effect on walking capacity (6-minute walk test) but no effect on UE function (Box & Block test) observed (n=777)
• Examining implementation of recommendations was planned to help explain results related to patient outcomes
19
Hypothesis
• The rate of implementing 18 selected treatment recommendations would improve from pre- to post-intervention to a greater extent in the active, facilitated KT intervention group than in the passive KT intervention group
20
Measurement of Implementation of 18 SCORE Treatment Recommendations
• Self-reported implementation (yes/no) using checklists
• PTs, OTs, nurses asked to complete after every patient seen in a 2-week period, pre- & post-intervention
• Intervention period: 16 months
21
Warning – This is a test! In the SCORE-IT study…
Present Level of Clustering
Clusters of providers within hospitals?
Clusters of patients within providers?
Clusters of measurements within
patients? 22
Analysis plan for each of 18 outcomes• Logistic Regression with random effects modeling (can account
for clustering)
• Tested for clustering effect at site, provider and patient levels
• Estimated the rate of adherence to each treatment recommendation pre- and post-intervention
• Compared the change from pre- to post-intervention between groups using an interaction term: group x time
• Analysis conducted with and without adjustment for clustering and covariates (eg patient motor function)
23
Significant clustering effects at Provider (67%) and Patient (39%) levels only
Treatment Site Provider Patient
1. Sit-to-stand Yes Yes2. LE ROM and/or stretching Yes3. Use of LE external support Yes Yes4. Task-specific training (i.e., stairs) Yes5. Training for sitting balance Yes6. Training for standing balance Yes7. FES for the LE
8. Walking practice Yes Yes9. Treadmill walking practice
10. UE ROM and/or stretching Yes Yes11. Interventions to prevent shoulder pain Yes Yes12. Task-specific training (i.e., self-care tasks) Yes Yes13. Techniques to reduce hand edema Yes14. Ice/heat or soft tissue massage for shoulder
15. FES for wrist/ arm/shoulder
16. Educate patient or caregiver on how to handle
arm or shoulderYes
17. UE constraint-induced therapy
18. Visual imagery to enhance arm recovery Yes
24
Between-group analysis – UE treatments: No effect before/after adjustment for clusteringTreatment Effect (ChangeE-ChangeC)
% (95% CI)
Effect
Regression Estimate (p-value)
Unadjusted Adjusted for
Clustering
Adjusted for Clustering
and Covariates
UE ROM/Stretching 5.2 (-4.4, 14.7) 0.1 (0.876) 0 (0.968)
Interventions to prevent shoulder
pain
4.9 (-5.1, 14.9) 0.8 (0.343) 0.8 (0.269)
Task-specific UE training 6.0 (-5.4, 17.3) 0.5 (0.237) 0.5 (0.237)
Techniques to reduce hand
edema
0 (-9.6, 8.8) -0.1 (0.734) -0.1 (0.759)
Shoulder ice/heat/massage 4.8 (-2.3, 11.8) 0.8 (0.487) 0.9 (0.432)
FES for UE 0.5 (-4.1, 5.1) DNC 1.7 (0.382)
Educate patient/caregiver 3.4 (-3.9, 10.6) 0.5 (0.718) 0.2 (0.916)
Constraint-induced therapy 1.8 (-5.6, 9.2) -0.8 (0.596) -0.9 (0.606)
Visual imagery 3.5 (-3.7, 10.8) 0.4 (0.381) 0.5 (0.338)
25
Between-group analysis – LE-Posture-Mobility Treatments: 2 of 5 effects no longer significant after adjustment for clustering
TreatmentEffect (ChangeE-ChangeC)
% (95% CI)
Effect
Regression Estimate (p-value)
UnadjustedAdjusted for
Clustering
Adjusted for
Clustering and
Covariates
Sit to stand 21.4 (10.4, 32.4) 1.1 (0.035) 1.0 (0.028)
Walking practice 22.0 (11.2, 32.8) 1.0 (0.029) 1.0 (0.043)
LE ROM/Stretching -14.7 (-26.0, -3.5) -1.1 (0.093) -1.1 (0.106)
Training for sitting
balance -14.6 (-28.2, -1.0) -0.8 (0.125) -0.9 (0.123)
Training for
standing balance -22.8 (-39.4, -6.2) -1.0 (0.040) -1.1 (0.037)
26
Take home messages
• Self-report methods to evaluate provider behaviour are common in KT intervention studies
• Many methods to evaluate provider behaviour:• Consider whether method captures practice in a range patients
over time, vulnerability to bias, feasibility, and cost
• Understand how limitations influence interpretation of results
• Plan to adjust for important covariates (patient, provider, hospital) based on theoretical or empirical evidence
• Qualitative and quantitative research helps us understand what influences provider practice 27
Campbell et al 2012; Campbell et al 2007
Take home messages
• Identify levels of clustering in design phase; review reporting guidelines for cluster RCTs
• In context of stroke rehabilitation guideline adoption:• Effect of patients clustered within provider has strongest
impact on estimated use of best practices, followed by data clustered within patients
• Hospital site does not appear influential
• Consult with a statistician re: impact of clustering and covariate adjustment on sample size estimation; plan to evaluate clustering in analysis
28
SCORE-IT Research TeamMark Bayley, MD, Toronto Rehabilitation Institute-University Health Network
Sharon Wood-Dauphinee, PhD, McGill University
Susan Barreca, PT, MSc
Sandra Black, MD, Sunnybrook Health Sciences Centre
Lucie Brosseau, PhD, University of Ottawa
Johanne Desrosiers, PhD, Université de Sherbrooke
Janice Eng, PhD, University of British Columbia
Ian Graham, PhD, University of Ottawa
Jeremy Grimshaw, PhD, University of Ottawa
Margaret Harrison, PhD, Queen’s University
Jeffery Hoch, PhD, University of Toronto
Susan Jaglal, PhD, University of Toronto
Nicol Korner-Bitensky, PhD, McGill University
Marilyn MacKay-Lyons, PhD, Dalhousie University
Nancy Mayo, PhD, McGill University
Carol Richards, PhD, Laval University
Monica Taljaard, PhD, Ottawa Health HealthResearch Institute
Robert Teasell, MD, Western University
Merrick Zwarenstein, MBBCh, PhD, Western University
SCORE-IT Coordinators:
Amanda Hurdowar
Jennifer Mokry
Olga Yaroslavtseva
Marion Hau
29
SCORE-IT Acknowledgements
We would like to acknowledge the hospitals and thank the many healthcare professionals and people with stroke who participated
Funding Participants
Thank you!
Nancy Salbach, PT, [email protected]
@nancysalbachKnowledge to Action Lab: http://www.physicaltherapy.utoronto.ca/research/ktalab/
32
CONSORT Diagram
33
Clustering EffectsTreatment Intracluster Correlation Coefficient
Site Provider Patient
1. Sit-to-stand 0.06 0.26* 0.10*
2. LE ROM and/or stretching 0.00 0.30* 0.04
3. Use of LE external support 0.11 0.10* 0.23*
4. Task-specific training (i.e., stairs) 0.00 0.35* 0.00
5. Training for sitting balance 0.01 0.16* 0.09
6. Training for standing balance 0.02 0.23* 0.08
7. FES for the LE 0.12 0.00 0.00
8. Walking practice 0.02 0.31* 0.11*
9. Treadmill walking practice 0.03 0.19 0.17
10. UE ROM and/or stretching 0.08 0.32* 0.06*
11. Interventions to prevent shoulder pain 0.08 0.16* 0.21*
12. Task-specific training (i.e., self-care tasks) 0.05 0.28* 0.06*
13. Techniques to reduce hand edema 0.06 0.06 0.22*
14. Ice/heat or soft tissue massage for
shoulder 0.14 0.16 0.20
15. FES for wrist/ arm/shoulder 0.24 0.16 0.04
16. Educate patient or caregiver on how to
handle arm or shoulder 0.05 0.25* 0.07
17. UE constraint-induced therapy 0.11 0.19 0.14
18. Visual imagery to enhance arm recovery 0.00 0.31* 0.04
*Significant clustering effect at Provider (67%) and Patient (39%) levels only.
Cluster sizes at the site, provider and patient level by study group and sampling time pointCluster Cluster Size Mean (SD)
Level 7 Treatments Implemented by RNs, OTs, and PTs
11 Treatments Implemented by OTs and PTs
Facilitated Passive Facilitated Passive
Pre Post Pre Post Pre Post Pre Post
Site:
Providers per site
15 (8) 13 (6) 10 (7) 13 (7) 6 (3)* 6 (3)* 5 (4)* 7 (4)*
Provider:
Patients per provider
3 (3) 2 (1) 2 (1) 2 (1) 2 (1) 2 (1) 2 (1) 2 (1)
Patient:
Forms per patient
13 (13) 7 (3) 6 (4) 6 (3) 4 (2) 4 (1) 4 (2) 3 (1)
*Smaller provider clusters per site for treatments that fewer health professional groups were asked
to implement.
35
Baseline Implementation of Recommended Treatments for LE/Mobility Function (unadjusted)
Treatment % of Times Implemented (95% CI) Level
n Facilitated KT n Passive KT
Sit to stand 647 20.4 (17.3, 23.5) 193 36.3 (29.5, 43.1) >15%
LE ROM/Stretching 151 15.9 (10.1, 21.7) 118 8.5 (3.4, 13.5) >15%
Use of external LE support 647 7.3 (5.3, 9.3) 193 15.0 (10.0, 20.1) <15%
Task-specific LE training 151 31.8 (24.3, 39.2) 118 26.3 (18.3, 34.2) >15%
Training for sitting balance 151 23.8 (17.0, 30.7) 118 17.0 (10.2, 23.7) >15%
Training for standing balance 151 51.7 (43.7, 59.6) 118 36.4 (27.7, 45.1) >15%
FES for LE 151 0.7 (-0.6, 2.0) 118 0 (0, 0) <15%
Walking practice 647 15.9 (13.1, 18.7) 193 31.6 (25.0, 38.2) >15%
Treadmill walking 151 2.7 (0.1, 5.2) 118 6.8 (2.2, 11.3) <15%36
Within-Group Unadjusted Change: Improved (green) and reduced (red) for LE/MobilityTreatment No.
forms
Pre to Post Change in % of Times Implemented (95% CI)
Facilitated KT Passive KT
Sit to stand 1381 18.7 (12.2, 25.3) -2.7 (-11.5, 6.2)
LE ROM/Stretching 547 -5.4 (-13.1, 2.3) 9.3 (1.1, 17.5)
Use of external LE support 1381 1.4 (-2.5, 5.3) 2.3 (-4.5, 9.1)
Task-specific LE training 547 6.7 (-4.3, 17.6) 11.5 (0.1, 22.9)
Training for sitting balance 547 -6.4 (-15.6, 2.9) 8.2 (-1.8, 18.2)
Training for standing balance 547 0.8 (-10.7, 12.2) 23.6 (11.6, 35.6)
FES for LE 547 0 (-1.9, 1.9) 0.7 (-0.7, 2.2)
Walking practice 1381 23.2 (16.8, 29.6) 1.2 (-7.4, 9.9)
Treadmill walking 547 -1.3 (-4.5, 2.0) -1.6 (-7.5, 4.3)
37
Baseline Implementation of Recommended Treatments for UE Function (unadjusted)
Treatment % of Times Implemented (95% CI) Level
n Facilitated KT n Passive KT
UE ROM/Stretching 647 12.7 (10.1, 15.2) 193 21.8 (15.9, 27.6) >15%
Interventions to prevent
shoulder pain647 25.0 (21.7, 28.4) 193 25.4 (19.2, 31.5) >15%
Task-specific UE training 647 28.9 (25.4, 32.4) 193 37.3 (30.5, 44.1) >15%
Techniques to reduce hand
edema151 7.3 (3.1, 11.4) 118 10.2 (4.7, 15.6) <15%
Ice/heat or soft tissue massage
for shoulder151 1.3 (-0.5, 3.2) 118 8.5 (3.4, 13.5) <15%
FES for UE 151 2.0 (-0.2, 4.2) 118 2.5 (-0.3, 5.4) <15%
Educate patient/caregiver how
to handle UE647 8.8 (6.6, 11.0) 193 13.0 (8.2, 17.7) <15%
Constraint-induced therapy 151 4.6 (1.3, 8.0) 118 10.2 (4.7, 15.6) <15%
Visual imagery 151 2.7 (0.1, 5.2) 118 5.1 (1.1, 9.1) <15%
38
Within-Group Unadjusted Change: Improved (green) and reduced (red) implementation for UETreatment No.
forms
Pre to Post Change in % of Times
Implemented
(95% CI)
Facilitated KT Passive KT
UE ROM/Stretching 1381 8.7 (3.2, 14.2) 3.5 (-4.3, 11.4)
Interventions to prevent shoulder
pain1381 0.7 (-5.5, 6.8) -4.3 (-12.1, 3.6)
Task-specific UE training 1381 12.0 (5.3, 18.8) 6.1 (-3.0, 15.2)
Techniques to reduce hand edema 547 -1.7 (-7.3, 3.9) -1.3 (-8.6, 6.0)
Ice/heat or soft tissue massage for
shoulder547 1.5 (-1.8, 4.7)
-3.3 (-9.6, 3.0)
FES for UE 547 -0.6 (-3.5, 2.4) -1.1 (-4.6, 2.4)
Educate patient/caregiver how to
handle UE1381 0.6 (-3.5, 4.7) -2.8 (-8.8, 3.2)
Constraint-induced therapy 547 -3.9 (-7.6, -0.3) -5.7 (-12.2, 0.8)
Visual imagery 547 3.6 (-1.1, 8.4) 0.1 (-5.4, 5.6)
39