Session 4: Analysis and reporting Managing missing data Rob Coe (CEM, Durham) Developing a statistical analysis plan Hannah Buckley (York Trials Unit) Panel on EEF reporting and data archiving Jonathan Sharples, Camilla Nevill, Steve Higgins and Andrew Bibby
37
Embed
Session 4: Analysis and reporting Managing missing data Rob Coe (CEM, Durham) Developing a statistical analysis plan Hannah Buckley (York Trials Unit)
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Session 4: Analysis and reporting
Managing missing dataRob Coe (CEM, Durham)
Developing a statistical analysis planHannah Buckley (York Trials Unit)
Panel on EEF reporting and data archivingJonathan Sharples, Camilla Nevill, Steve Higgins and Andrew Bibby
Managing missing data
Rob Coe
EEF Evaluators Conference, York, 2 June 2014
∂
The problem
Only if everyone responds to everything is it still a randomised trial– Any non-response (post-randomisation) → not an RCT
It may not matter (much) if– Response propensity is unrelated to outcome– Non-response is low
Lack of ‘middle ground’ solutions– Mostly people either ignore or use very complex stats
3
∂
What problem are we trying to solve?
We want to estimate the distribution of likely effects of [an intervention] in [a population]– Typically represented by an effect size and CI
Missing data may introduce bias and uncertainty– Point estimate effect size different from observed– Probability distribution for ES (CI) widens
4
What kinds of analysis are feasible to reduce the risk of bias from missing data?
5
∂
Vocabulary
Missing Completely at Random (MCAR)– Response propensity is unrelated to
outcome
Missing at Random (MAR)– Missing responses can be perfectly
predicted from observed data
Missing Not at Random (MNAR)– We can’t be sure that either of the
above apply
6
Ignore missingness
Statistics:IWP, MI
??
∂
“When data are missing not at random, no method of obtaining unbiased estimates exists that does not incorporate the mechanism of non-random missingness, which is nearly always unknown. Some evidence, however, shows that the use of a method that is valid under missing at random can provide some reduction in bias.”
Bell et al, BMJ 2013
7
∂
Recommendations1. Plan for dealing with missing data should be in
protocol before trial starts
2. Where attrition likely, use randomly allocated differential effort to get outcomes
3. Report should clearly state the proportion of outcomes lost to follow up in each arm
4. Report should explore (with evidence) the reasons for missing data
5. Conduct simple sensitivity analyses for strength of relationship between
Outcome score and missingness
Treatment/Outcome interaction and missingness
8
∂
If attrition is not low (>5%?)
6. Model outcome response propensity from observed variables
7. Conduct MAR analyses• Inverse weighted probabilities• Multiple imputation
8. Explicitly evaluate plausibility of MAR assumptions (with evidence)
9
∂
10
∂
11
∂
Useful references Bell, M. L., Kenward, M. G., Fairclough, D. L., & Horton, N. J.
(2013). Differential dropout and bias in randomised controlled trials: when it matters and when it may not. BMJ: British Medical Journal, 346:e8668. http://www.bmj.com/content/346/bmj.e8668
Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual review of psychology, 60, 549-576.
National Research Council. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press, 2010. http://www.nap.edu/catalog.php?record_id=12955
Shadish, W. R., Hu, X., Glaser, R. R., Kownacki, R., & Wong, S. (1998). A method for exploring the effects of attrition in randomized experiments with dichotomous outcomes. Psychological Methods, 3(1), 3.