Designing More Accessible Designing More Accessible Achievement Tests for All Achievement Tests for All Students Students Stephen N. Elliott Learning Sciences Institute and Department of Special Education Vanderbilt University CCSSO 2009 National Conference on Student Assessment
20
Embed
Designing More Accessible Achievement Tests for All Students Stephen N. Elliott Learning Sciences Institute and Department of Special Education Vanderbilt.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Designing More AccessibleDesigning More Accessible Achievement Tests for All Students Achievement Tests for All Students
Stephen N. Elliott Learning Sciences Institute
andDepartment of Special Education
Vanderbilt University
CCSSO 2009 National Conference on Student Assessment
CCSSO NSA Conference / Elliott 2009
2
Projects & PartnersProjects & Partners• CAAVES: Consortium for Alternate Assessment Validity and Experimental Studies
– USDE funded; 2006-2009 – Partners: AZ, HI, ID, & IN + Vanderbilt Measurement Group +
Discovery Education Assessment
• CMAADI: Consortium for Modified Alternate Assessment Development and Implementation– USDE funded; 2007-2010– Arizona Dept. of Education– Indiana Dept. of Education
• Visit Websites for Resources Discussed Today– http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home.xml– http://peabody.vanderbilt.edu/LSI_Projects/C-MAADI_Project_Home.xml– http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home/TAMI_Project.xml
• Students with disabilities who exhibit persistent academic difficulties.– Inattention– Organizational difficulties– Poor reading fluency– History of below proficient
test performances– Low self-efficacy with
testing
CCSSO NSA Conference / Elliott 2009
4
Key TermsKey Terms• AccessAccess: the opportunity for test-takers to demonstrate proficiency on
the target construct of a test or a test item. In essence, complete access is manifest when a test-taker is able show the degree to which he/she knows the tested content. Access, therefore, must be understood as an interaction between individual test-taker characteristics and features of the test itself.
• AccommodationAccommodation: widely recognized in state testing guidelines as individualized changes to the setting, scheduling, presentation format, or response format of an assessment.
• ModificationModification: alterations or adjustments of test items to facilitate access for virtually all test takers. Appropriate modifications ….– Remove extraneous material,
– Maintain the same depth of knowledge (DOK),
– Do NOT change the grade-level construct being measured, and
– Increase the validity of the inference from the test score.
CCSSO NSA Conference / Elliott 2009
5
Anatomy of an ItemAnatomy of an Item
CCSSO NSA Conference / Elliott 2009
6
CAAVES Procedures CAAVES Procedures
We completed the following…..
• Modified a common set of existing reading and math items to create items designed to be more accessible and still measure the same grade-level content as the original items.
• Conducted a cognitive lab study with a small sample of students with and without disabilities to gain their insights into which item modifications are preferred and most likely to improve test access for students whose disability involves reading difficulties.
• Conducted a cross-state experimental study to compare the effects of tests with and without modified items on students’ test performances and test score comparability.
• Conducted post-assessment survey of all students concerning their perceptions of item types and cognitive ease.
CCSSO NSA Conference / Elliott 2009
7
Guiding Theories & ResearchGuiding Theories & Research
• Evidenced-based model of test score validity,
• Universal design principles,
• Cognitive Load Theory for designing instructional materials, and
• Item writing research and practices.
CCSSO NSA Conference / Elliott 2009
8
Examples of Theory-Guided & Examples of Theory-Guided & Data-Based Item ModificationsData-Based Item Modifications
CCSSO NSA Conference / Elliott 2009
9
Example: Original to Modified ItemExample: Original to Modified Item
CCSSO NSA Conference / Elliott 2009
10
Overview of ResultsOverview of Results
Elliott, et al. (in press), Exceptional Children
Modifications Benefited all GroupsModifications Benefited all Groups
CCSSO NSA Conference / Elliott 2009
11Elliott, et al. (in press), Exceptional Children
CCSSO NSA Conference / Elliott 2009
1212
Item Summary Reports: An ExampleItem Summary Reports: An Example
Estimating Impact
•Will AA-MASs result in more students with disabilities being considered “proficient” for AYP?
• We have explored the impact of some hypothetical cut scores for the CAAVES Reading and Math Scores.
•An actual Standard Setting is needed.
CCSSO NSA Conference / Elliott 2009
13
Elliott, et al. (in press), Exceptional Children
CCSSO NSA Conference / Elliott 2009
14
Method for Documenting OTLMethod for Documenting OTL
AZ Cog LabStudy, 2008
CCSSO NSA Conference / Elliott 2009
15
Cognitive LabsCognitive Labs
Excerpted from Kettler, Elliott, & Beddow, in press Peabody Journal of Education
CCSSO NSA Conference / Elliott 2009
Post-Assessment Focus GroupsPost-Assessment Focus Groups