Interpreting Translational Research Findings Incredible Years Conference, Cardiff March 9 th , 2011 Christopher Whitaker, Senior Statistician, NWORTH Tracey Bywater, Research Fellow, School of Psychology
Feb 14, 2016
Interpreting Translational Research Findings
Incredible Years Conference, Cardiff March 9th, 2011
Christopher Whitaker, Senior Statistician, NWORTHTracey Bywater, Research Fellow, School of
Psychology
Overview• Translational research & complex interventions• How do we/should we report or interpret results• Welsh Sure Start RCT of parent programme &
outcome measures• Methods of assessing change:
– Means & Standard deviations– Effect sizes– Numbers needed to treat
• Summary & conclusions
What is translational research?Translational research transforms scientific discoveries arising from laboratory, clinical, or population studies into clinical applications to tackle all sorts of disorders/diseases etc
Translational Research Working Group: www.cancer.gov/researchandfunding/trwg/TRWG-definition-and-TR-continuum
Complex interventions
• EVALUATION – “to strengthen or empower”, more recently it is defined as an assessment of value.
• Should we look at end outcome only or ‘how we got there’?
• Social policy interventions, delivered in education, public health practice, or family and children services, are complex interventions (Medical Research Council (MRC), 2009).
• Complex interventions comprise several interacting components
Selected dimensions of complexity according to MRC (2009): implications for development and evaluation
Number of components and interactions between them - theoretical understanding is needed of how the intervention causes change, so that weak links in the causal chain can be identified and strengthened Number and difficulty of behaviour changes required by those delivering or receiving the intervention - a thorough process evaluation is needed to identify implementation problems lack of impact may reflect implementation failure rather than genuine ineffectivenessNumber and variability of outcomes - a single primary outcome may not be most appropriate, a range of measures may be required
Levels of evidence
1. Expert opinion The developer says
2. Case series Observe IY recipients
3. RCT Randomly assign to IY or TAU
Randomisation 1:1 ratio
Welsh Sure Start StudyHutchings et al (2007)
• Parenting intervention in Sure Start services for children at risk of developing conduct disorder: pragmatic randomised controlled trial
• Children aged 3-4 years, randomised 2:1• Targeted population – over cut off on Eyberg
Child Behaviour Inventory – Intensity 7-point scale, 36-252, cut off 127– Problem scale – yes/no answers, 0-36, cut off 11
Measures
Measures were administered at baseline, 6, 12, and 18 months post baseline. They included (amongst others):
• Kendall Self Control Rating Scale (Kendall & Wilcox, 1979)
• Conners Hyperactivity Questionnaire (Conners, 1994)
• Strengths & Difficulties Questionnaire (Goodman, 1997)
ECBI-I Follow up
TAU 144.0
(n = 49)
IY 122.3
(n = 104)
ECBI mean at 6-month (1st) follow up
ECBI-I Baseline Follow up
TAU 141.3 144.0
(n = 49)
IY 146.8 122.3
(n = 104)
ECBI mean at 6-month (1st) follow up and baseline
IY mean = 122.3, TAU mean = 144.0
ECBI-I Baseline Follow up
TAU 141.3 144.0
(n = 49) (26.8) (33.0)
IY 146.8 122.3
(n = 104) (27.0) (35.1)
ECBI mean and SD at 6-month (1st) follow up and baseline
IY mean = 122.3, TAU mean = 144.0
ECBI-I Baseline Follow up BL - FU
TAU 141.3 144.0 +2.7
(n = 49) (26.8) (33.0)
IY 146.8 122.3 -24.5
(n = 104) (27.0) (35.1)
-27.2
ECBI mean and SD at 6-month (1st) follow up, baseline and change
Conclusion – IY lowers ECBI by 27.2 points on the scale
• NO – 27.2 is an approximate value• Statistical analysis - gives a more precise value• Take account of each participants
1. Baseline value2. Sure start area
• Statistical analysis finds IY lowers ECBI by 25.05 points on the scale
Better summary
• IY lowers ECBI by 25.05 points on the scale• 95% Confidence Interval (CI) for this mean is
14.92 to 35.18
• Based on this sample of data we are 95% confident that the effect of IY is to reduce ECBI between 14.92 and 35.18 points on the scale
95% CI for other measures
mean Lo CI Hi CI Significance
ECBI-I 25.05 14.92 35.18 p < .001
ECBI-P 4.42 2.00 6.85 p < .001
Conners 3.39 1.47 5.31 p < .001
Kendall SRCS 8.16 0.68 15.61 p = .033
SDQ total 1.52 -0.24 3.28 p = .091
Normal distribution plots of data
Normal distribution plots of artificial data (SD = 5)
Conclusion from the plots
• Differences in the means are the same• SD is different• Lots of overlap suggests lesser effect• Can we measure overlap
• Difference in means relative to SD
Effect size
• Uses the mean difference• Uses the variability of the mean difference
(SD)• Is comparable between the measures• How to calculate effect size – different ways,
we use Cohen’s d:• d = (IY mean – TAU mean) / SD
Which measure has IY had biggest effect on
Effect size Lo CI Hi CI Significant
ECBI-I 0.89 0.54 1.24 p < .001
ECBI-P 0.63 0.28 0.98 p < .001
Conners 0.61 0.27 0.96 p < .001
Kendall SCRS 0.38 0.03 0.73 p = .033
SDQ total 0.30 -0.05 0.65 p = .091
At baseline all children have ECBI-I >= 127 OR ECBI-P >= 11
At (1st) Follow up
Number Below both cut-offs
Benefit (%)
TAU 49 9 18%
IY 104 38 37%
Idea behind Number Needed to Treat
• With IY 37% benefit, with TAU 18% benefit• In 6 families with IY approximately 2 benefit• In 6 families with TAU approximately 1
benefits
• So in 6 families 1 more benefits with IY than with TAU
Number Needed to Treat (NNT)Calculation
• 38/104 benefit with IY• 9/49 benefit with TAU• Difference is 38/104 – 9/49 = 0.1817
• NNT = 1 / 0.1817 = 5.5• NNT is the number of families that need to be
treated with IY rather than TAU for one additional family to benefit
Attrib
utab
le R
isk R
educ
tion
(%) 10
20
30
40
0
-10
2.5
3.3
5
10
10
NNT
NNT 5.5 (72.5, 3.1)
NUMBER NEEDED to TREAT
Benefit
Summary & Conclusions• Be clear on
– What research question is being asked– What service managers/policy makers want to know and why
• Ensure sensitive validated measures are used• Identify most useful method of presenting data for target
audience, e.g. in this case– Mean values are sensitive to change but not easy to interpret,
SD & other factors should be taken in to account– Effect sizes are derived from means and shows magnitude of
change– NNT is not very sensitive but useful to give guidance on
numbers required to reduce prevalence rates & therefore costs
ReferencesConners, C. K. (1994). The Conners Rating Scales: Use in clinical assessment, treatment
planning and research. In M. Maruish (Ed.), Use of Psychological Testing for Treatment Planning and Outcome Assessment. Hillsdale, New Jersey: Erlbaum.
Eyberg, S. M. (1980). Eyberg Child Behavior Inventory. Journal of Clinical Child Psychology, 9, 27.
Goodman, R. (1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology, Psychiatry, and Allied Disciplines, 38 (5), 581-586.
Hutchings, J., Bywater, T., Daley, D., Gardner, F., Whitaker, C., Jones, K., Eames, C. & Edwards, R. T. (2007) Parenting intervention in Sure Start services for children at risk of developing conduct disorder: pragmatic randomised controlled trial. British Medical Journal, 334, 678-682. Accessible at: http://www.bmj.com/content/334/7595/678.full
Kendall, P. & Wilcox, L. (1979). Self-control in children: Development of a rating scale. Journal of Consulting and Clinical Psychology, 47, 1020-1029.
Medical Research Council (2009). Developing and Evaluating Complex Interventions: New guidance. Accessible at: www.mrc.ac.uk/complexinterventionsguidance
Additional reading
• Effect sizes– Cohen, J. (1988). Statistical Power for the Behavioural
Sciences. Erlbaum, Hillsdale, NJ, USA.
• Calculating the Number Needed to Treat (Altman & Anderson, 1999) Accessible at:– http://www.bmj.com/content/319/7223/1492.full
• Confidence Intervals for the difference between 2 proportions:– http://faculty.vassar.edu/lowry/prop2_ind.html