1 !"#"$ &’ ("))*+ TAMI TEST ACCESSIBILITY AND MODIFICATION INVENTORY TM Quantifying and Improving the Accessibility of Tests and Test Items CCSSO 2009 National Conference on Student Assessment
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
!"#"$%&'%("))*+
TAMITEST ACCESSIBILITY AND MODIFICATION INVENTORYTM
Quantifying and Improving the Accessibility of Tests and Test Items
CCSSO 2009 National Conference on Student Assessment
Beddow
Overview
1) Definition of accessibility;
2) Test Accessibility and Modification Inventory (TAMI);
a) Evolution and application of the instrument;
b) Guiding influences;
c) Content and Structure of the TAMI and TAMI
Accessibility Rating Matrix (ARM);
3) Directions for future research.
2
Beddow
Accessibility• Accessibility is the extent to which a product,
environment, or system eliminates barriers and permits equal access to all components and services for all individuals. (Beddow, Kettler, & Elliott, 2008)
• Test accessibility is the extent to which a test and its constituent item set permits the test-taker to demonstrate knowledge of the target construct . Thus, an accessible test:
1. Eliminates barriers;
2. Permits equal access to all components and features for all test-takers; and
3. Yields scores from which subsequent inferences do not reflect error that is the result of incomplete test-taker access.
3
Beddow
Accessibility is an Interaction
• Additionally, accessibility involves an interaction between characteristics of the test and individual test-taker characteristics.
• An item may permit one individual to access the target construct with minimal effort, whereas for another individual, the same item may require the expenditure of essential cognitive resources to gain access the target construct.
• Both individuals may be equally knowledgeable of the tested content, but accessibility issues may preclude one from demonstrating what he or she knows.
• The evaluation of test and test item accessibility requires familiarity not only with the tested material but also with the target population of a test.
4
Beddow
Test-Taker Characteristics
• In the case of tests for students with special needs (e.g., students identified with disabilities), characteristics of the test-taker population include:
• Inattention;
• Organizational difficulties;
• Poor reading fluency;
• History of below-proficient test performances;
• Low test self-efficacy.
5
Beddow
Accessibility is a Degree
• Just as validity is discussed in terms of degrees, a test is never deemed entirely accessible or entirely inaccessible.
• Most - if not all - tests can be improved with respect to accessibility.
• The TAMI and TAMI Accessibility Rating Matrix (ARM) can be used to guide determinations about which aspects of test items may be improved to enhance their accessibility.
• Ultimately, accessibility evaluations should be based on a firm understanding of universal design principles, cognitive load theory, and research on test and item development.
6
Beddow
2% Alternate Assessment
7
Data from Altman, J. R., Lazarus, S. L.,
Thurlow, M. L., Quenemoen, R. F.,
Cuthbert, M., & Cormier, D. C. (2008).
2007 survey of states: Activities, changes,
and challenges for special education.
Minneapolis, MN: University of Minnesota,
National Center on Educational
Outcomes.
Beddow 8
Grade 8 Sample Item
AA-MASResponse FrequencyResponse FrequencyResponse FrequencyResponse Frequency Item InformationItem InformationItem InformationItem Information
AA-MAS
Eligibility A B C* D p DOK R # Words
Not eligible 16% 50% 30% 4% 0.30
2 8.7 66
Eligible 25% 36% 21% 15% 0.21
2 8.7 66
*key; R = Readability Score
Beddow
TAMI: Evolution
9
Project Instrument Purpose Influences
Consortium for
Alternate Assessment
Validity and
Experimental Studies
(CAAVES)
Item Accessibility and
Modification Guide
(IAMG)(Beddow, Kettler, & Elliott,
2007)
Checklist to sensitize
assessment professionals
from four states (AZ, ID, IN,
HI) to research and
theoretical principles for
modifying a set of 8th-grade
reading and math items.
•NCEO guidance(e.g., Thompson, Johnstone, & Thurlow, 2002; Johnstone, Thurlow, Moore, & Altman, 2006)
•Item-writing research(e.g., Haladyna, Downing, & Rodriguez, 2002)
Consortium for
Modified Alternate
Assessment
Development and
Implementation
(CMAADI); Arizona
Test Accessibility and
Modification Inventory
(TAMI)(Beddow, Kettler, & Elliott,
2008)
Inventory to train a group of
item-writers from AZ to use a
systematic procedure based
on the anatomy of a test item
to enhance the accessibility
of items across multiple
grade levels.
•Cognitive Load Theory(e.g., Chandler & Sweller, 1991)
•Universal Design for Assessment(e.g., Dolan & Hall, 2001)
•Web Accessibility Guidance(e.g., http://www.w3.org)
Consortium for
Modified Alternate
Assessment
Development and
Implementation
(CMAADI); Indiana
TAMI Accessibility
Rating Matrix (ARM)(Beddow, Elliott, & Kettler,
2009)
Dual set of rubrics to conduct
an accessibility review of a
large pool of test items
across mutliple content
areas, grade levels, and item
types.
•TAMI validity studies;
•CAAVESCognitive LabPilot Study
•CMAADICognitive LabPilot Study / Focus Group
Beddow
TAMI:Theory and Development
10
Beddow
TAMI: Overview
11
• The Test Accessibility and Modification Inventory (TAMI; Beddow, Kettler, & Elliott, 2008) was developed as a decision-making tool to facilitate the analysis of new and existing tests and test items with the purpose of enhancing their accessibility.
• The TAMI was influenced by four primary areas of study:
1) Universal design principles;
2) Cognitive load theory;
3) Research on test and item development; and
4) Guidance on web and computer accessibility.
Beddow
• Universal design, as defined in the Assistive Technology Act (P.L.
105-394, 1998), is “a concept or philosophy for designing and
delivering products and services that are usable by people with the
widest possible range of functional capabilities, which include
products and services that are directly usable (without requiring
assistive technologies) and products and services that are made
usable with assistive technologies” (§3(17)).
• The use of universal design principles reduces the need for many
specific kinds of assistive technology devices and assistive technology
services by building in accommodations for individuals with
disabilities before rather than after production. The use of universal
design principles also increases the likelihood that products (including
services) will be compatible with existing assistive technologies. These
principles are increasingly important to enhance access to
information technology, telecommunications, transportation, physical
structures, and consumer products (PL105-394(§3(10)); emphasis
added).
12
Universal Design
Beddow
Cognitive Load Theory
13
“Considering the wide variety of
different variables that have
been studied...there seems to be
some limitation built into us
either by learning or by the
design of our nervous systems,
a limit... [on] our channel
capacities...” (Miller, 1956, p.86)
Beddow
Cognitive Load Theory
14
To properly gain knowledge from
instruction, students must:
1.Attend to presented
material;
2.Mentally organize the
material into a coherent
structure; and
3.Integrate the material with
existing knowledge.
Beddow
Cognitive Load Theory
15
Intrinsic Load
Amount of mental
processing
requisite for
completing a task.
Germane Load
Cognitive demand
that is not
necessary for
gaining essential
knowledge but
enhances learning
through
automation or
generalization.
Extraneous Load
Demand for
cognitive
resources to attend
to and integrate
nonessential
elements that are
preliminary to
actual learning.
“Intrinsic, extraneous, and germane cognitive loads are additive in that,
together, the total load cannot exceed the working memory resources
available if learning is to occur” (Paas, Renkl, and Sweller, 2003, p.2).
+ +
Beddow
• Split attention effect (Chandler & Sweller, 1996)
• Redundancy effect (Mayer & Moreno, 2003)
• Seductive detail effect (Graves et al., 1988; 1991; Wade, Schraw, Buxton, & Hayes, 1993; Goetz & Sadoski, 1995)
16
Cognitive Load Theory
(Mayer & Moreno, 2003)
Beddow
Test & Item Writing
Guidelines
17
Downing, Haladyna, and
Rodriguez (2002)
synthesized test & item-
writing research into 31
guidelines for writing
multiple-choice items.
Beddow
TAMI:Content and Structure
18
Beddow
Anatomy of an ItemStimulus
19
Visual(pictures, graphs, tables,
charts, or figures)
Stem
Answer Choices
key (B) and
distractors (A & C)Page Layout
X
X
Beddow
TAMI: Overview
20
The TAMI consists of two components:
1. The original Inventory:
a) Item Analysis (51 descriptors); and
b) Computer-Based Test Analysis (35 descriptors)
2. The Accessibility Rating Matrix (ARM).
Beddow
TAMI ARM: StructureThe TAMI Accessibility Rating Matrix (ARM) contains two rubrics.
Item Analysis (i.e., item element) rubric:
Passage / Item Stimulus
Item Stem
Visuals
Answer Choices
Page / Item Layout
Overall Analysis (i.e., overall item-level) rubric.
21
4 Maximally Accessible for Nearly All Test-Takers
3 Maximally Accessible for Most Test-Takers
2 Maximally Accessible for Some Test-Takers
1 Inaccessible for Many Test-Takers
ARM
Accessibility
Levels
Beddow
ARM: Item Analysis22
Beddow 23
ARM Item Analysis
Beddow 24
ARM Overall Analysis
Beddow 25
ARM Record Form
Beddow
TAMI:Application
26
Beddow 27
Grade 8 Sample Item
AA-MAS
Eligibility
Response FrequencyResponse FrequencyResponse FrequencyResponse Frequency Item InformationItem InformationItem InformationItem InformationAA-MAS
Eligibility A B C* D p DOK R # Words
Not eligible 16% 50% 30% 4% 0.30
2 8.7 66
Eligible 25% 36% 21% 15% 0.21
2 8.7 66
*key; R = Readability Score
Beddow 28
AA-MASEligibility
Response FrequencyResponse FrequencyResponse FrequencyResponse Frequency Item InformationItem InformationItem InformationItem InformationAA-MASEligibility A B C* D p DOK R # Words
Not eligible 16% 50% 30% 4% 0.302 8.7 66
Eligible 25% 36% 21% 15% 0.212 8.7 66
*key; R = Readability Score
1
2
X
3
2
2
XX
X
XX
X
X
XX
X
X
X XXX
X
5
Overall Rating:
Maximally
Accessible for
Some Test-Takers
Beddow 29
• Reduced extraneous verbiage in stimulus and stem.
• Added bold to key words.
• Added a visual.• Reduced the number of
distractors.• Increased white space.
Original Modified
Beddow 30
Original ItemAA-MAS Response FrequencyResponse FrequencyResponse FrequencyResponse Frequency Item InformationItem InformationItem InformationItem Information
Eligibility A B C* D p DOK R # Words
Not eligible 16% 50% 30% 4% 0.302 8.7 66
Eligible 25% 36% 21% 15% 0.212 8.7 66
AA-MAS Response FrequencyResponse FrequencyResponse Frequency Item InformationItem InformationItem InformationItem Information
Eligibility A B C* p DOK R # Words
Not eligible 16% 45% 36% 0.362 5.6 57
Eligible 29% 36% 35% 0.352 5.6 57
Modified Item
DOK: No change
Readability: -3.1
# of Words: -9 (!14%)Original! Modified!
Not Eligible!
Eligible!
Difficu
lty (p
)
Condition
1
*key; R = Readability Score
Beddow
Tests can be improved for students with and without special needs!
31
Beddow 33
ReferencesAltman, J. R., Lazarus, S. L., Thurlow, M. L., Quenemoen, R. F., Cuthbert, M., & Cormier, D. C. (2008). 2007 survey of
states: Activities, changes, and challenges for special education. Minneapolis, MN: University of Minnesota,
National Center on Educational Outcomes.
Beddow, P. A., Elliott, S. N., & Kettler, R. J. (2009). TAMI Accessibility Rating Matrix (ARM). Nashville, TN: Vanderbilt
University.
Beddow, P. A., Kettler, R. J., & Elliott, S. N. (2008). Test Accessibility and Modification Inventory (TAMI). Nashville, TN:
Vanderbilt University.
Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8, 293-332.
Dolan, R. P., & Hall, T. E. (2001). Universal design for learning: Implications for large-scale assessment. IDA Perspectives,
27, 22-25.
Goetz, E. T., & Sadosky, M. (1995). The perils of seduction: Distracting details or incomprehensible abstractions? Reading
Research Quarterly, 30, 500-511.
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for
classroom assessment. Applied Measurement in Education, 15(3), 309-344.
Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38,
43-52.
Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information.
The Psychological Review, 63, 81-97.
Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent Developments. Educational
Psychologist, 38, 1-4.
Rodriguez, M. C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research.
Educational Measurement: Issues and Practice, 24, 3-13.
Thurlow, M. L., Altman, J. R., & Vang, M. (In press). Annual Performance Reports: 2006-2007 State Assessment Data.
Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
Wade, S. E., Schraw, G., Buxton, W. M., & Hayes, M. T. (1993). Seduction of the strategic reader: Effects of interest on
strategies and recall. Reading Research Quarterly, 28, 3-24.
Web Accessibility Initiative (2008). Web Content Accessibility Guidelines (WCAG) Version 2.0. Available from www.w3.org
Related Documents
