Robert Dolan Tracey Hall Developing Accessible Tests with Universal Design and Digital Technologies.

Robert DolanRobert DolanTracey HallTracey Hall

Developing Accessible Tests Developing Accessible Tests with Universal Design and with Universal Design and

Digital TechnologiesDigital Technologies

Restatement of the Problem:Restatement of the Problem:How to Standardize Assessment of How to Standardize Assessment of Nonstandard (i.e. Nonstandard (i.e. AllAll) Individuals?) Individuals?

• The challengeThe challengeEnsuring adequate individualization of test delivery Ensuring adequate individualization of test delivery without compromising comparabilitywithout compromising comparability

• Proposed solutions:Proposed solutions:– Application of universal design principlesApplication of universal design principles– Use of digital technologiesUse of digital technologies

Universal DesignUniversal Design

Originated in architecture as a means for Originated in architecture as a means for ensuring environments, and later products ensuring environments, and later products

and communications, be designed from and communications, be designed from the start to consider the needs of the the start to consider the needs of the

widest possible array of userswidest possible array of users(Ron Mace, 1996)(Ron Mace, 1996)

MatthewMatthew

Guggenheim MuseumGuggenheim Museum

Universal DesignUniversal Design

• NotNot one-size-fits-all, but rather one-size-fits-all, but rather provides users with provides users with alternativesalternatives..

• Designed in from the beginning, Designed in from the beginning, not added on later.not added on later.

• Increases access opportunities for Increases access opportunities for everyoneeveryone..

Universal Design for LearningUniversal Design for Learning

• Universal Design for Learning (UDL) Universal Design for Learning (UDL) extends universal design from a extends universal design from a physical space to a pedagogical space.physical space to a pedagogical space.

• New approach for teaching, learning, New approach for teaching, learning, and assessment that draws on new and assessment that draws on new brain research and new media brain research and new media technologies to respond to individual technologies to respond to individual learner differences.learner differences.

Universal Design for LearningUniversal Design for Learning

• UDL curricula, teaching practices, and UDL curricula, teaching practices, and policies are inherently flexible so as to policies are inherently flexible so as to support individual learning differences.support individual learning differences.

• Reduces the demand on educators to Reduces the demand on educators to develop and implement modifications develop and implement modifications and accommodations.and accommodations.

• Supports individual learner differences Supports individual learner differences through use of:through use of:– Multiple means of representation– Multiple means of expression– Multiple means of engagement

• Relies on scaffolds and accommodations Relies on scaffolds and accommodations to support students’ challenges and thus to support students’ challenges and thus improve access to learning.improve access to learning.

Universal Design for LearningUniversal Design for Learning

Multiple Means of RepresentationMultiple Means of RepresentationTo give learners various ways ofTo give learners various ways of

acquiring information and knowledgeacquiring information and knowledge

• Provide multiple examplesProvide multiple examples• Highlight critical features Highlight critical features • Provide multiple media and formats Provide multiple media and formats • Support background contextSupport background context

Multiple Means of ExpressionMultiple Means of ExpressionTo provide learners alternatives for To provide learners alternatives for

demonstrating what they knowdemonstrating what they know

• Provide flexible models of skilled performance Provide flexible models of skilled performance • Provide opportunities to practice with Provide opportunities to practice with

supports supports • Provide ongoing, relevant feedback Provide ongoing, relevant feedback • Offer flexible opportunities for demonstrating Offer flexible opportunities for demonstrating

skillskill

Multiple Means of EngagementMultiple Means of EngagementTo tap into learners' interests, challenge them To tap into learners' interests, challenge them

appropriately, and motivate them to learnappropriately, and motivate them to learn

• Offer choices of content and tools Offer choices of content and tools • Offer adjustable levels of challenge Offer adjustable levels of challenge • Offer choices of rewards Offer choices of rewards • Offer choices of learning contextOffer choices of learning context

Role of Digital TechnologyRole of Digital Technology

New media, such as digital text, New media, such as digital text, sound, images, and the Web,sound, images, and the Web,can be adjusted for different can be adjusted for different individuals and can reduce individuals and can reduce

construct-irrelevant barriers,construct-irrelevant barriers,thus opening doors to learning. thus opening doors to learning.

http://www.cast.org/teachingeverystudent/mlk

- Folktales Example -- Folktales Example -

Universal Design for Learning Universal Design for Learning and Assessmentand Assessment

Universal Design for Learning Universal Design for Learning principles apply globally across principles apply globally across

standards, instruction, materials, standards, instruction, materials, and assessments.and assessments.

FormativeEvaluation• Analyze Errors• Monitor Progress• Modify Instruction

Instructional Delivery• Access• Participation and Engagement • Monitor performance and pace • Immediate Feedback

Instructional Design• Determine Content• Instructional Strategies• Materials Selection and Sequence• Select examples• Provide for cumulative review

Initial Assessment• Archival Assessment• Formal Standardized Measures• Diagnostic Assessments

Multiple means of: • Representation• Expression• Engagement

UDL

UDL & Instructional / Assessment CycleUDL & Instructional / Assessment Cycle

Universal Design & AccommodationsUniversal Design & Accommodations

• Accommodations critical to reduce construct-Accommodations critical to reduce construct-irrelevant variance.irrelevant variance.

• Universal design requires that accommodations Universal design requires that accommodations be be considered from the startconsidered from the start to ensure to ensure effectiveness.effectiveness.

• Ideally accommodations should be seamlessly Ideally accommodations should be seamlessly integrated across integrated across instruction and assessmentinstruction and assessment..

The goal is to reduce accommodations The goal is to reduce accommodations asas retrofitsretrofits..

Universal Design and Universal Design and AssessmentAssessment

Current EffortsCurrent EffortsTwo complementary efforts:Two complementary efforts:

– Improving current state of testing through Improving current state of testing through application of original Mace principles.application of original Mace principles.

– Improving future of testing by increasing Improving future of testing by increasing flexibility and individualization usingflexibility and individualization usingdigital technologies.digital technologies.

UDA Directed ResearchUDA Directed Research

Compare and contrast two states, Compare and contrast two states, Kentucky and Maine,Kentucky and Maine,

with respect to universal designwith respect to universal designand technology-based testing.and technology-based testing.

Kentucky Online SystemKentucky Online System

Technology Skills ChecklistTechnology Skills Checklist

• Basic computer skillsBasic computer skills• Keyboarding skillsKeyboarding skills• Word processing skillsWord processing skills• Text-reader / screen-reader skillsText-reader / screen-reader skills• Online assessment skillsOnline assessment skills

Fleming, Kearns, Dethloff, Lewis & Dolan JSET 2006Fleming, Kearns, Dethloff, Lewis & Dolan JSET 2006

Research on Universal Design Research on Universal Design and Assessmentand Assessment

• Limited research to date.Limited research to date.• Accommodations research in Accommodations research in

general shows mixed results.general shows mixed results.• Technology-based testing Technology-based testing

research in general shows mixed research in general shows mixed results.results.

Keyboarding vs. HandwritingKeyboarding vs. Handwriting

• Study by Boston College Lynch School Study by Boston College Lynch School of Education (Michael Russell & Walt of Education (Michael Russell & Walt Haney) and Wellesley, MA, Public Haney) and Wellesley, MA, Public Schools (Tom Plati).Schools (Tom Plati).

• Significant improvement in MCAS Significant improvement in MCAS scores for typically-achieving 4th, 8th, scores for typically-achieving 4th, 8th, and 10th graders’ when using and 10th graders’ when using keyboarding for open-response keyboarding for open-response questions.questions.

Keyboarding vs. HandwritingKeyboarding vs. Handwriting

Study of Universally Designed Study of Universally Designed Technology-Based TestingTechnology-Based Testing

• Pilot study with high-school Pilot study with high-school students with learning disabilities.students with learning disabilities.

• Test of civics and U.S. history.Test of civics and U.S. history.• Paper-and-pencil vs. computer-Paper-and-pencil vs. computer-

based testing with optionalbased testing with optionaltext-to-speech read-aloudtext-to-speech read-aloud(CBT-TTS).(CBT-TTS).

Prototype CBT SystemPrototype CBT System

Effect of CBT-TTS on Test ScoresEffect of CBT-TTS on Test Scores

Dolan, Hall, et Dolan, Hall, et al. JTLA 2005al. JTLA 2005

Research Take Home PointsResearch Take Home Points

• Both technology and universal Both technology and universal design promising for reducing design promising for reducing construct-irrelevant variance.construct-irrelevant variance.

• Importance of matching Importance of matching instruction and assessment instruction and assessment techniques and technologies.techniques and technologies.

• Additional research needed!Additional research needed!

Future of Universal Design, Future of Universal Design, Technology, and AssessmentTechnology, and Assessment

• Providing students with choice Providing students with choice during testing.during testing.

• Developing richer, more authentic Developing richer, more authentic assessment situations.assessment situations.

• Progress monitoring.Progress monitoring.

Providing Students with Providing Students with Choice During TestingChoice During Testing

What is the effect of providing students What is the effect of providing students choice of equivalent questions /stimuli?choice of equivalent questions /stimuli?(i.e. (i.e. multiple means of representationmultiple means of representation))

Mei Ling gave the following description of a three-dimensional figure.

• The solid has 6 faces• The solid has 8 vertices.• The solid has 12 edges.

8. Which of the following figures matches Mei Ling’s description? Click on any figure to see it spin in 3D.

Developing Richer, MoreDeveloping Richer, MoreAuthentic AssessmentsAuthentic Assessments

Can universal design and Can universal design and technology be used to develop technology be used to develop

assessment items that reduce not assessment items that reduce not only physical and sensory barriers only physical and sensory barriers

but cognitive barriers as well?but cognitive barriers as well?

Progress MonitoringProgress Monitoring

Can technology improveCan technology improveformative assessments throughformative assessments through

real-time feedback and individualization real-time feedback and individualization of instruction?of instruction?

Commercial Tools available Commercial Tools available for Progress Monitoringfor Progress Monitoring

• AIMSWEBAIMSWEB• DIBELSDIBELS

Progress Monitoring Built in a Supported Writing Environment

Technology Limitations & ConcernsTechnology Limitations & Concerns

• Availability of technology and the Availability of technology and the infrastructure to support it.infrastructure to support it.

• Need to retain traditional administration Need to retain traditional administration modes for use under certain situations.modes for use under certain situations.

• Potential for inequities between students.Potential for inequities between students.• Comparability issues.Comparability issues.• Students’ ability to exercise choice wisely.Students’ ability to exercise choice wisely.

ConclusionConclusion

• Technology can help realize the Technology can help realize the flexibility necessary to accurately flexibility necessary to accurately and fairly assess diverse students.and fairly assess diverse students.

• The principles of Universal Design The principles of Universal Design for Learning can inform the for Learning can inform the appropriate use of technology in appropriate use of technology in assessment.assessment.