-2 -1 0 1 2 3 4 5 6 7 8 9 10 0 20 40 60 80 PERFORMANCE (N-BACK LEVEL)) AGE (IN YEARS) PERFORMANCE GAINS BY AGE Baseline Performance Average Performance in last three sessions Gain in Performance Baseline (2 nd Order Poly) Last three sessions (2 nd Order Poly) Gain in performance (2 nd Order Poly) Individual Differences in Working Memory Training Shafee Mohammed 1 ; Benjamin Katz 2 ; Chelsea M. Parlett 1 ; Martin Buschkuehl 3 ; and Susanne M. Jaeggi 1 University of California, Irvine 1 , University of Michigan 2 , MIND Research Institute 3 INTRODUCTION RESEARCH QUESTIONS • How do individual difference factors, such as age, baseline ability, and n-back training type (spatial vs. verbal), influence the outcome of training? • How does one’s pattern of performance on the training task influence transfer gains on untrained tasks? • Data: • 418 participants from varied age range, education level, and geographic locations. • At least 15 sessions of working memory training. Anguera et al. (2012). Behavioral Brain Research. Buschkuehl et al. (2014). Cognitive, Affective, and Behavioral Neuroscience. Jaeggi et al. (2011). Proceedings of the National Academy of Sciences of the United States of America. Jaeggi et al. (2011). Paper presented at the Eighteenth Annual Cognitive Neuroscience Society Meeting. Jaeggi et al. (2014). Memory and Cognition. Jaeggi et al. (2008). Proceedings of the National Academy of Sciences of the United States of America. Jaeggi et al. (2010). Intelligence. Jonides et al. (2010). Presented at the Office of Naval Research - Contractor's Meeting, Arlington, VA. Katz et al. (under review). Seidler et al. (2010). Technical Report No. M-CASTL 2010-01, University of Michigan, Ann Arbor. Stepankova et al. (2014). Developmental Psychology. Tsai et al. (in preparation) Zhang et al. (2014). 26th Annual Convention of the Association for Psychological Science, San Francisco, CA. METHOD RESULTS • Age, country in which the study was conducted, and baseline abilities predict one’s ability to improve at WM training. • Gender, N-back type, and supervision have no significant effect. • Prediction accuracy is significantly different from chance with nearly 76% (0.02) (200 iterations) on an average. DISCUSSION CONCLUSIONS • Baseline characteristics, and age are important factors. • Age, and baseline performance, are good predictors of training performance (R 2 = 0.30). • Further analysis will include non-linear mixed effect models and cluster analyses. • Working memory is critically important for success in school and at work (Gathercole et al., 2003; Higgins et al., 2007). • One’s performance on an adaptive and challenging longitudinal working memory intervention may serve as a useful assay to investigate cognitive plasticity. Error Matrix Target Label Predicted Label Count 0 0 34 0 1 10 1 1 33 1 0 12 Accuracy: 76% (0.02) Authors’ note: MB is employed at MIND Research Institute, whose interest is related to this work. SMJ has an indirect financial interest in the MIND Research Institute. REFERENCES Gathercole, S. E., Brown, L., & Pickering, S. J. (2003). Working memory assessments at school entry as longitudinal predictors of National Curriculum attainment levels. Educational and Child Psychology, 20(3), 109-122. Higgins, D. M., Peterson, J. B., Pihl, R. O., & Lee, A. G. (2007). Prefrontal cognitive ability, intelligence, Big Five personality, and the prediction of advanced academic and workplace performance. Journal of Personality and Social Psychology, 93(2), 298. Jaeggi, S. M., Buschkuehl, M., Shah, P., & Jonides, J. (2014). The role of individual differences in cognitive training and transfer. Memory & Cognition, 42(3), 464-480. PREDICTORS OF TRAINING • Not every person improves equally on a WM training task. • Identifying the factors that potentially contribute to explaining such individual differences is the key to improving training. • Data mining and machine learning can be viable approaches to investigating plasticity. Figure 2. Average performance on the first three sessions(blue), performance on last three sessions (gray), and performance gain (purple) as a function of Age. Figure 1. An example showing visual and auditory 2-back tasks. The stimuli are presented chronologically from left to right as a visual stimulus or an auditory stimulus respectively. Participants should match the current stimulus with the n th stimulus (2 nd in this case.) Table 1. The accuracy of the algorithm to detect whether or not a subject is in the top fifty percentile. Table 4. Values in the descriptive table are raw scores and standardized units for regression. Numbers in bold are statistically significant (in this case, p<0.01). Subjects aged 10-20, from US, Male, Supervised (as opposed to Unsupervised), Trained on spatial n-back, are left out as controls. Descriptive statistics Regression Results Sample Characteristics Mean SD Beta SE Age (Range 7-78 years) 22.51 16.11 Age Squared 765.68 1313.59 Age Centered 0 16.11 0.25 0.01 Age Centered Squared 258.89 627.57 -0.48 0.00 Gender (Female Proportion) 0.51 0.02 0.08 Baseline N-Back Level 2.64 0.95 0.11 0.05 Final N-Back level 3.74 1.64 Gain 1.11 1.044 Training Characteristics Supervised participants 0.43 0.08 0.01 Location (US Participants) 0.81 -0.23 0.01 N-back Type -0.03 0.01 Spatial 0.35 Verbal 0.11 Object Total Single N-back 0.60 Total Dual N-back 0.40 Total Observations 418 R-Squared 0.300 Gathering the data and cleaning Descriptive statistical analysis Data mining methods – Logistic classifier and Decision tree analysis Non-linear mixed models (planned) Nearest neighbor analysis (planned) Class Probability 1 0.715 0 0.678 0 0.870 0 0.567 1 0.502 0 0.503 Table 2. The prediction probability of each individual’s accuracy to detect whether or not each subject is in the top fifty percentile. The individual’s in the bold have a low predictive probabilities. Figure 3. The prediction accuracies over 200 iterations. Attained prediction range is 0.69 - 0.81. -2 0 2 4 6 0 10 20 30 40 50 60 70 80 N-back level Age in years Training gains based on training locations Non-US Participants US Participants Figure 4. The difference between the US (in red) and non-US (blue) participant training gains. The individuals with negative post training gains are exclusively from the US. N-Back Level Age Figure 7. (right) shows a decision tree model used to predict and classify individuals. Each individual starts at the purple dot and moves along the branches of the tree based on a decision choice for each condition. Only a single branch is shown in full detail for illustration purposes. Figure 5. (left) The baseline performance (top graph) and the gains (bottom graph) of the top 50% of the participants (in green) and bottom 50% of the participants (in red). True Positive Rate False Positive Rate Schema Number of examples 315 Number of features 7 Number of unpacked features 7 Number of classes 2 Model Properties Number of Trees 10 Max tree depth 6 Training accuracy 0.92 Validation Accuracy 0.85 Training log-loss 0.27 Validation log-loss 0.39 Figure 6.(left) ROC curve obtained by aggregating all predictions from 200 repetitions. Table 3.(above) Shows the model characteristics and the validation estimates. DATA SOURCES