Measurement and analysis of provider behaviour change in knowledge translation research: Is there a best practice? 2 nd National Knowledge Translation Conference in Rehabilitation, April 19, 2018 Nancy Salbach, PT, PhD CIHR New Investigator in Knowledge Translation Associate Professor, Department of Physical Therapy, University of Toronto
39
Embed
Measurement and analysis of provider behaviour change in ...
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Measurement and analysis of provider behaviour change in knowledge translation research: Is there a best practice?
2nd National Knowledge Translation Conference in Rehabilitation, April 19, 2018
Nancy Salbach, PT, PhDCIHR New Investigator in Knowledge Translation
Associate Professor, Department of Physical Therapy, University of Toronto
Presentation Objectives:
1. To consider the advantages and limitations of methods for measuring provider behaviour change following a KT intervention
2. To understand which quantitative approaches to measuring provider behaviour change may lead to clustered data and how to evaluate clustering in data analysis
2
Outcomes of KT studies in RehabilitationSystematic review: Jones et al 2015
3
More than 75% of KT intervention studies* in rehab target professional or process outcomes
• Provider practice = primary outcome as patient function influenced by practices of “other” providers
5
Most professional-level outcomes measured by provider self-report
Systematic review: Jones et al 2015
6
Warning! Brain squeeze
What are the pros and cons of different methods for measuring behaviour change in rehabilitation providers following KT interventions?
7
https://imgflip.com/meme/58388563/
Thinking-hard
Provider Self-report
Pros
• Feasible
• Inexpensiveif online
• Data on large group
Cons
• Measurement at one point in time
• Attrition over time
• Social desirability bias
• Item reliability?
8
Patient Self-report
Pros
• Inexpensive if online
• Can capture practice over time
Cons
• Privacy - Difficulty obtaining patient emails
• Recall
• Expense - phone/mail survey
• Low response rate
• Is response linked to health?9
Interviews & Focus Groups
Pros
• Deep understanding of mechanisms of behaviour change
Cons
• Interviews=resource intensive
• Focus groups: scheduling and opportunity to speak
• Limited for large groups10
Observing Provider Practice
Pros
• Observer unbiased
• Range of patients
Cons
• Expensive, resource intensive
• Representativeness depends on duration of observation
• Observation may influence practice11
https://www.smartcherry.in/your-
observation-power-smart-cherry/
Health record review
Pros
• Retrospective: Less vulnerable to bias
• Range of patients
Cons
• Inconsistent documentation
• Expensive, resource intensive
• If prospective, providers can change documentation behaviour
12
http://www.bcm
j.org/mds-
be/merit-and-
agony-
retrospective-
chart-reviews-
medical-
student%E2%8
0%99s-
perspective
Now a word about clustering
• Clustering is when people or data in your study form natural groupings because the people or data within a group are more related to each other than they are to people or data outside the group
• Can you think of examples of clustering in rehabilitation?
13
Clustering of Providers within HospitalsEg. Guideline implementation study, examine provider use of recommendations in 3 rehab hospitals
14
Hospital 1 Hospital 2 Hospital 3
Clusters
Clustering of Patients within ProvidersEg. Guideline implementation study, examine provider use of recommendations with each patient
15
Provider 1 Provider 2
Patients Patients
Provider 3
Patients
Clusters
Clustering of Measurements within PatientsEg. Guideline implementation study, examine provider use of recommendations across sessions with same patients
16
Patient 1 Patient 2
Data points Data points
Patient 3
Data points
Clusters
Multi-level Clustering
Data
points
Hospital
Hospital 2 Hospital 3Cluster Level
Patients
Provider
What happens when you don’t account for clustering?
• It depends on whether clustering has an influence
• If clustering influences your study outcome, and you don’t account for clustering in the analysis, you will likely get the wrong answer
• Let’s take a look at which level of clustering was important in a guideline implementation trial
18
Stroke Canada Optimization of Rehabilitation through Evidence-Implementation Trial (SCORE-IT)
• National cluster randomized guideline implementation trial
• 20 stroke inpatient rehabilitation centres randomized to a multi-component facilitated or passive KT intervention for guideline implementation
• Primary outcome was patient function: Effect on walking capacity (6-minute walk test) but no effect on UE function (Box & Block test) observed (n=777)
• Examining implementation of recommendations was planned to help explain results related to patient outcomes
19
Hypothesis
• The rate of implementing 18 selected treatment recommendations would improve from pre- to post-intervention to a greater extent in the active, facilitated KT intervention group than in the passive KT intervention group
20
Measurement of Implementation of 18 SCORE Treatment Recommendations
• Self-reported implementation (yes/no) using checklists
• PTs, OTs, nurses asked to complete after every patient seen in a 2-week period, pre- & post-intervention
• Intervention period: 16 months
21
Warning – This is a test! In the SCORE-IT study…
Present Level of Clustering
Clusters of providers within hospitals?
Clusters of patients within providers?
Clusters of measurements within
patients? 22
Analysis plan for each of 18 outcomes• Logistic Regression with random effects modeling (can account
for clustering)
• Tested for clustering effect at site, provider and patient levels
• Estimated the rate of adherence to each treatment recommendation pre- and post-intervention
• Compared the change from pre- to post-intervention between groups using an interaction term: group x time
• Analysis conducted with and without adjustment for clustering and covariates (eg patient motor function)
23
Significant clustering effects at Provider (67%) and Patient (39%) levels only
Treatment Site Provider Patient
1. Sit-to-stand Yes Yes2. LE ROM and/or stretching Yes3. Use of LE external support Yes Yes4. Task-specific training (i.e., stairs) Yes5. Training for sitting balance Yes6. Training for standing balance Yes7. FES for the LE
8. Walking practice Yes Yes9. Treadmill walking practice
10. UE ROM and/or stretching Yes Yes11. Interventions to prevent shoulder pain Yes Yes12. Task-specific training (i.e., self-care tasks) Yes Yes13. Techniques to reduce hand edema Yes14. Ice/heat or soft tissue massage for shoulder
15. FES for wrist/ arm/shoulder
16. Educate patient or caregiver on how to handle
arm or shoulderYes
17. UE constraint-induced therapy
18. Visual imagery to enhance arm recovery Yes
24
Between-group analysis – UE treatments: No effect before/after adjustment for clusteringTreatment Effect (ChangeE-ChangeC)