This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Running head: DIAGNOSTIC UTILITY SCATTER 1
Evidence-based assessment and Cognitive Profile Scatter: Clinical Acumen or Clinical
Illusion?
Ryan J. McGill
The College of William and Mary
Author Note
Paper presented at the 2017 annual meeting of the Southeastern Psychological Association, Atlanta, GA.
Standardization data from the Kaufman Assessment Battery for Children, Second Edition
Correspondence concerning this paper should be addressed to Ryan J. McGill, School of Education, The College of William and Mary, P. O. Box 8795 Williamsburg, VA. 23187. E-Mail: [email protected]
DIAGNOSTIC UTILITY SCATTER 2
Abstract
Within the professional literature, it is frequently suggested that interpretation of cognitive
profile scatter may be useful for clinical and/or diagnostic decision-making. To wit, Hale and
colleagues (2008) posit that cognitive scatter is a defining characteristic of learning disabilities
and that individuals with learning disabilities may have higher levels of scatter than normal
controls. To investigate the tenability of this claim, the present study employed diagnostic
efficiency statistics and other psychometric methods for examining the utility of proposed
diagnostic indicators (e.g., receiver operative characteristic curve, Bayesian nomogram) to
determine the degree to which cognitive scatter accurately discriminated between individuals
with and without a known learning disability diagnosis in the Kaufman Assessment Battery for
Canivez, G. L. (2013). Psychometric versus actuarial interpretation of intelligence and related aptitude batteries. In D. H. Saklofske, C. R. Reynolds, & V. L. Schwean (Eds.). The Oxford handbook of child psychological assessment (pp. 84-112). New York: Oxford University Press. Flanagan, D. P., & Alfonso, V. C. (Eds.). (2010). Essentials of specific learning disability
identification. Hoboken NJ: Wiley. Hale, J. B., & Fiorello, C. A. (2004). School neuropsychology: A practitioner’s handbook. New
York: Guilford. Hale, J. B., Fiorello, C. A., Dumont, R., Willis, J. O., Rackley, C, & Elliot, C. (2008).
Differential Ability Scales-Second Edition (neuro)psychological predictors of math performance for typical children and children with math disabilities. Psychology in the Schools, 45, 838-858. doi: 10.1002/pits.20330
Hunsely, J., & Mash, E. J. (2007). Evidence-based assessment. Annual Review of Clinical
Psychology, 3, 29-51. doi: 10.1146/annurev.clinpsy.3.022806.091419 Katz, D. L. (2001). Clinical epidemiology & evidence-based medicine. Thousand Oaks, CA:
Sage. Kaufman, A. S. (1994). Intelligent testing with the WISC-III. New York: John Wiley. Kaufman, A. S., & Kaufman, N. L. (2004a). Kaufman Assessment Battery for Children (2nd ed.).
Circle Pines, MN: American Guidance Service. Kaufman, A. S., & Kaufman, N. L. (2004b). Kaufman Assessment Battery for Children manual
(2nd ed.). Circle Pines, MN: American Guidance Service. Kavale, K. A. (2002). Discrepancy models in the identification of learning disability. In R.
DIAGNOSTIC UTILITY SCATTER 13
Bradley, L. Donaldson, & D. Hallahan (Eds.), Identification of learning disabilities: Research to practice (pp. 369-426). Mahwah, NJ: Erlbaum.
Kavale, K. A., & Forness, S. R. (1984). A meta-analysis of the validity of Wechsler scale
profiles and recategorizations: Patterns or parodies? Learning Disability Quarterly, 7, 136-156. doi: 10.2307/1510314
Kessel, J. B., & Zimmerman, M. (1993). Reporting errors in studies of the diagnostic
performance of self-administered questionnaires: Extent of the problem, recommendations for standardized presentation of results, and implications for the peer review process. Psychological Assessment, 5, 395-399. doi: 10.1037/1040-3590.5.4.395
Kranzler, J. H., Floyd, R. G., Benson, N., Zaboski, B., & Thibodaux, L. (2016). Classification
agreement analysis of cross-battery assessment in the identification of specific learning disorders in children and youth. International Journal of School & Educational Psychology, 4, 124-136. doi: 10.1080/21683603.2016.1155515
Lilienfeld, S. O., Ammirati, R., & David, M. (2012). Distinguishing between science
pseudoscience in school psychology: Science and scientific thinking as safeguards against human error. Journal of School Psychology, 50, 7-36. doi: 10.1016/j.jsp.2011.09.006
MacCallum, R. C., Zhang, S., Preacher, K. J., Rucker, D. D. (2002). On the practice of
dichotomization of quantitative variables. Psychological Methods, 7, 19-40. doi: 10.1037/1082-989X.7.1.19
Macmann, G. M., & Barnett, D. W. (1997). Myth of the master detective: Reliability of
interpretations for Kaufman’s “intelligent testing” approach to the WISC-III. School Psychology Quarterly, 12, 197-234. doi: 10.1037/h0088959
DIAGNOSTIC UTILITY SCATTER 14
McFall, R. M., & Treat, T. A. (1999). Quantifying the information value of clinical assessments
with signal detection theory. Annual Review of Psychology, 50, 215-241. doi: 10.1146/annurev.psych.50.1.215
McGill, R. J. (2016). Invalidating the full scale IQ score in the presence of significant factor
score variability: Clinical acumen or clinical illusion? Archives of Assessment Psychology, 6 (1), 49-79.
McGill, R. J., Styck, K. S., Palomares, R. S., & Hass, M. R. (2016). Critical issues in specific
learning disability identification: What we need to know about the PSW model. Learning Disability Quarterly, 39, 159-170. doi: 10.1177/0731948715618504
Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow
progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806- 834. doi: 10.1037/0022-006X.46.4.806
Meehl, P. E., & Rosen, A. (1955). Antecedent probability and the efficiency of psychometric
Naglieri, J. A. (2000). Can profile analysis of ability test scores work? An illustration using the
PASS theory and CAS with an unselected cohort. School Psychology Quarterly, 15, 419- 433. doi: 10.1037/h0088798
Pfeiffer, S. I., Reddy, L. A., Kletzel, J. E., Schmelzer, E. R., & Boyer, L. M. (2000). The
practitioner’s view of IQ testing and profile analysis. School Psychology Quarterly, 15, 376-385. doi: 10.1037/h0088795
Prifitera, A., & Dersh, J. (1993). Base rates of WISC-III diagnostic subtest patterns among
normal, learning-disabled, and ADHD samples. Journal of Psychoeducational
DIAGNOSTIC UTILITY SCATTER 15
Assessment Monograph Series, 43-55.
Rapaport, D., Gil, M., & Schafer, R. (1945). Diagnostic psychological testing: The theory,
statistical evaluation, and diagnostic application of a battery of tests (Vol. 1). Chicago: Yearbook Medical.
Streiner, D. L. (2003). Diagnosing test: Using and misusing diagnostic and screening test.
Journal of Personality Assessment, 81, 209-219. doi: 10.1207/S15327752JPA8103_03 Stuebing, K. K., Fletcher, J. M., Branum-Martin, L., & Francis, D. J. (2012). Evaluation of the
technical adequacy of three methods for identifying specific learning disability. School Psychology Review, 41, 3-22.
Swets, J. A. (1988). Measuring the accuracy of diagnostic systems. Science, 240, 1285-1293.
doi: 10.1126/science.3287615 Watkins, M. W. (2000). Cognitive profile analysis: A shared professional myth. School
Psychology Quarterly, 15, 465-479. doi: 10.1037/h0088802 Watkins, M. W. (2002). Diagnostic Utility Statistics [Computer software]. Phoenix, AZ: Ed &
Psych Associates.
Watkins, M. W. (2003). IQ subtest analysis: Clinical acumen or clinical illusion? The Scientific
Review of Mental Health Practice, 2, 118-141.
Watkins, M. W. (2009). Errors in diagnostic decision making and clinical judgment. In T. B.
Gutkin & C. R. Reynolds (Eds.), Handbook of school psychology (4th ed.; pp. 210-229). New York, NY: Wiley.
Watkins, M. W., Glutting, J. J., & Youngstrom, E. A. (2005). Issues in subtest profile analysis.
In D. P. Flanagan & P. L. Harrison (Eds.), Contemporary intellectual assessment: Theories, tests, and issues (2nd ed.; pp. 251-268). New York: Guilford Press.
DIAGNOSTIC UTILITY SCATTER 16
Wiggins, J. S. (1988). Personality and prediction: Principles of personality assessment. Malabar,
FL: Krieger Publishing. Youngstrom, E. A. (2014). A primer on receiver operating characteristic analysis and diagnostic
efficiency statistics for pediatric psychology: We are ready to ROC. Journal of Pediatric Psychology, 39, 204-221. doi: 0.1093/jpepsy/jst062
Youngstrom, E. A., Choukas-Bradley, S., Calhoun, C. D., & Jensen-Doss, A. (2015). Clinical
guide to the evidence-based assessment approach to diagnosis and treatment. Cognitive and Behavioral Practice, 22, 20-35. doi: 10.1016/j.cbpra.2013.12.005
Youngstrom, E. A., & Van Meter, A. (2016). Empirically supported assessment of children and
adolescents. Clinical Psychology: Science and Practice, 23, 327-347. doi:
10.1111/cpsp.12172
DIAGNOSTIC UTILITY SCATTER 17
Table 1 Means and Standard Deviations of the KABC-II Index and FCI Scores for Participants Ages 6-18 in the Normative Sample Based Upon Known SLD Condition (N = 2,223) Non-SLD SLD KABC-II Score M SD M SD d Short-Term Memory (Gsm) 100.7 14.8 88.9 14.2 -0.81 Visual Processing (Gv) 100.5 14.8 90.7 13.5 -0.69 Long-Term Memory (Glr) 101.0 14.9 87.4 11.8 -1.01 Fluid Reasoning (Gf) 100.6 14.7 88.8 13.9 -0.82 Crystallized Ability (Gc) 100.6 14.7 88.6 13.8 -0.84 Fluid-Crystallized Index (g) 100.8 14.6 85.7 11.5 -1.14 Note. KABC-II = Kaufman Assessment Battery for Children-Second Edition (Kaufman & Kaufman, 2004b); g = general intelligence; Differences significant at p < .001 for all 6 score comparisons across groups based upon known SLD condition. Average cognitive profile scatter for the Non-SLD group (25.2) and SLD group (24.4) were both clinically significant.
DIAGNOSTIC UTILITY SCATTER 18
Table 2 Diagnostic Efficiency Statistics for Different Levels of Cognitive Scatter on the KABC-II Predicting the Presence of SLD Scatter Level AUC Sensitivity Specificity LR+ LR- PPV NPV DOR IPPP Accuracy Κ 10 Points .501 .964 .039 1.00 0.91 .050 .954 1.11 .000 .086 .000 15 Points .511 .882 .140 1.02 0.84 .051 .957 1.22 .001 .177 .002 23 Points .498 .576 .420 0.99 1.01 .049 .949 0.98 .000 .428 .000 30 Points .469 .243 .696 0.80 1.08 .040 .945 0.73 -.009 .673 -.018 Note. AUC = area under curve, LR+ = likelihood ratio for positive test results LR- = likelihood ratio for negative test results, PPV = positive predictive value, NPV= negative predictive value, DOR = diagnostic odds ratio, IPPP = incremental validity of positive test diagnosis (Hsu, 2002), Accuracy = agreement/hit rate, Κ = kappa coefficient for chance agreement (Streiner, 2003).
DIAGNOSTIC UTILITY SCATTER 19
Diagnostic Condition SLD No SLD Test Outcome Significant
Scatter True Positive False Positive
Type I Error Positive Predictive Value (PPV): Probability that scatter will be observed when SLD is present.
No Scatter False Negative
Type II Error True Negative Negative Predictive Value (NPV):
Probability that scatter is not present when SLD is not observed.
Sensitivity: Probability that
there will be scatter when SLD is identified.
Specificity: Probability that there will not be scatter when SLD is not identified.
Prevalence Rate: Base rate of SLD in the sample Accuracy: Rate at which true positives and true negatives are correctly identified False Alarm Rate (1-Specificity): Rate at which individuals present with scatter but do not have SLD Figure 1. Model for evaluating the degree to which scatter accurately predict specific learning disability (SLD) within a diagnostic decision framework.
DIAGNOSTIC UTILITY SCATTER 20
Figure 2. ROC graph illustrating the comparisons of true-positive and false-positive rates from individuals with and without SLD diagnoses form the KABC-II normative sample when all possible cut scores were used.
DIAGNOSTIC UTILITY SCATTER 21
10 Points
15 Points
23 Points
30 Points
Figure 3. Probability nomogram used to combine prior probability with likelihood ratios to estimate revised, posterior probability of SLD diagnosis using different levels of cognitive profile scatter as a positive or negative test.