Measured Progress ©2012 GAAP: Guidelines for Accessible Assessments Project June 26 th, 2014 CCSSO National Conference on Student Assessment New Orleans,

Measured Progress ©2012

GAAP: Guidelines for Accessible Assessments ProjectJune 26th, 2014

CCSSO National Conference on Student Assessment

New Orleans, LAThe content of this presentation were developed under a grant from the U.S. Department of Education. However, those contents do not necessarily represent the policy of the U.S. Department of Education and you should not assume endorsement by the Federal government.

• GAAP Project Background and Overview – Jen Higgins

• Research Design and Randomized Control Trial Findings – Lisa Famularo

• Cognitive Lab Findings – Vitaliy Shyyan

• Finalizing Audio and Sign Guidelines – Lisa Famularo

• Discussion, State and Consortia Perspective – Trinell Bowman and Deborah Matthews

Session Agenda

• Item Development• Appropriate complexity of content, language, diagrams, and

layout

• Delivery Interface• Intuitive, non-distracting, input device independent, flexible tool

support

• Universal Design Sets the Table for Access

Universal Design

Ability to concentrate on the problem is one of the

constituent parts of the successful mental work

Delivery System Accessibility Supports

We all see things the same way. We see words in groups or phrases.The print is more dominant than thebackground. The print shows nomovement. The printed letters areevenly black. Black print on whitepaper gives the best contrast foreveryone. White background lookswhite.

Delivery System Accessibility Supports

• Traditional image of a test item.

What is a Test Item?

Digital Item File

ASL tags

Braille Tags

Text-Based Default Form

Spoken (Read Aloud)Tags

Language Learner Tags

Translation

• Focus is to develop research based guidelines for the audio and sign representation of test items.

• Primary project activities:• Review of literature and existing state guidelines• Draft consensus based audio and sign guidelines

• Conduct consensus building webex sessions for audio guideline issues

• Conduct in-person meetings for sign guideline issues

• Conduct cognitive labs• Conduct randomized control trial• Revise and finalize guidelines

Project Overview

State Audio Guidelines for Assessment

Provide guidelines in

words

Show guideline applied to specific content

Show guidelines

applied to full sample items

Provide list of guidelines for

symbols, abbreviations,

etc.

Provide list of commonly

mispronounced words

Connecticut ✔

Delaware ✔ ✔

Georgia ✔ ✔

Hawaii ✔ ✔ ✔

Kansas ✔

Maryland ✔ ✔ ✔

Minnesota ✔ ✔ ✔ ✔ ✔

North Carolina ✔

Oregon ✔ ✔

Utah ✔

Virginia ✔ ✔

States (and consortia) are making progress in improving reliability of audio accessibility

•Seven states provide oral/reader scripts that human administrators use to read tests to one or more eligible students.

•Eleven states provide pre-recorded versions of test content on CD or cassette that accompany paper-based test materials.

•Three states provide a pre-programmed audio version of the test as part of their computer based test administration.

•Smarter Balanced and PARCC have developed audio guidelines that are being used to create accessible audio versions of CCSS test content.

State Audio Guidelines for Assessment

Framework for Audio Guidelines

• Project Background and Methodology

• Intended Use

• Item Example

• Strategies for Graphic and Image Description

• Use of Guidelines with Text-to-Speech

• Content Specific Guidelines with Examples

Audio Guideline Example

ParenthesesExample: 3(x + y) = 6 Audio GuidelineText Only/Text and Graphicsa) Read the parentheses by referring to the opening and closing of the parentheses using the language “open parenthesis” and “close parenthesis.”b) It is important to reference the close of the parentheses to be clear on when the parenthetical expression ends.c) When reading an equation or expression with multiple parts and sets of parentheses, pause to help differentiate between sections. d) Read brackets using the same language as parentheses, but with the word bracket (“open bracket” and “close bracket”). Application of Audio GuidelineExample 1:Three times open parenthesis x plus y close parenthesis equals six.

Graphic and Image Description

State Sign Guidelines for Assessment

• No states have content specific sign guidelines for assessment. Alaska, Massachusetts, and South Carolina provide a pre-recorded sign DVD for one or more grade levels/content areas.

• Existing state documents provide guidelines for:

• Qualification of sign language administrators

• Warnings against cueing and elaboration

• Use of “home signs” or “locally developed signs”

• Testing environment (lighting, group size, etc).

Framework for Sign Guidelines

• Introduction: rationale for guidelines, intended use

• Team: roles and responsibilities in developing videos

• Process: steps in creating high quality ASL videos

• ASL Grammar: review of ASL grammar considerations to create high quality videos

• Content Specific Guidelines (fingerspelling, equations, graphics and images, etc).

• Filming Considerations

ASL Video Example

• Identified areas of focus for the research during guideline development

• Cognitive labs to understand extent to which audio and sign representations developed according to the guidelines remove construct irrelevant barriers for students

• Randomized control trials (RCT) to understand impact on student performance

Research

RCT Research Areas: Sign

Research Area Grade 3-5

Grade 6-9

Grade 9-12

Finger Spelling ✔ ✔ ✔

Use of Space ✔

Plurality ✔

Equations ✔ ✔

Diamond (item-set up)

✔ ✔

RCT Research Areas: Audio

Research Area Grade 3-5

Grade 6-9

Grade 9-12

Equations ✔ ✔ ✔

Graphs ✔

Images ✔ ✔ ✔

Drag and drop ✔

Decimals ✔

RCT Sample Sizes

Grade Reading based disability

Low vision

ELL DHH No need

Total Audio Study

3-5 17 9 1 1 33 61

6-8 1 9 0 6 110 126

9-12 7 13 0 1 73 94

Total 25 31 1 8 216 281

Grade DHH Low RLow M

DHHLow RHigh M

DHHHigh RLow M

DHHHigh RHigh M

No need

Total Sign Study

3-5 20 18 4 12 107 161

6-8 68 16 8 7 96 195

9-12 92 27 2 5 45 171

Total 180 61 14 24 248 527

State Participation in GAAP

Note: The bolded states participated in the cognitive labs. The bolded states with an asterisk participated in both the cognitive lab and randomized trial.

States Audio Sign States Audio Sign

Alabama X X Maryland X X

Arizona* X Massachusetts X X

California X X Michigan X X

Colorado X X Minnesota* X

Connecticut X New Hampshire X

Delaware X X North Carolina X X

Florida X X Ohio X X

Georgia X Oregon* X X

Hawaii* X Pennsylvania X X

Idaho X X Texas X

Illinois X Utah X X

Indiana X Virginia X X

Kansas X X West Virginia X X

Kentucky X X

RCT Design

Item Form 1 Form 2 Form 31. Introductory Item Audio/Sign Audio/Sign Audio/Sign

2. Issue 1 Item 2 Audio/Sign Version 1 Audio/Sign Version 2 No Audio/Sign






8. Issue 1 Item 3 Audio/Sign Version 2 No Audio/Sign Audio/Sign Version 1






14. Issue 1 Item 1 No Audio/Sign Audio/Sign Version 1 Audio/Sign Version 2






• Did general education students perform higher on the research test forms than students with sign and audio access needs? Yes.

• ASL Study (p<.001)

• Audio Study (p<.001)

Randomized Control Trial Findings

n Mean Std. Dev.

General Ed 248 10.25 4.19

DHH 279 4.59 2.66

n Mean Std. Dev.

General Ed 216 10.27 4.33

Audio Need 65 6.94 3.67

• Did students with audio and sign needs performed better on supported items than they did on unsupported items? Yes.

• ASL Study (p=.013)

• Audio Study (p=.025)


Mean n Std. Dev.

Std. Error Mean

Pair 1 Score on supported items

3.24 279 2.04 .12

Weighted score on unsupported items

2.91 279 2.28 .14

Mean n Std. Dev.

Std. Error Mean

Pair 1 Score on supported items

5.00 65 2.71 .34

Weighted score on unsupported items

4.25 65 3.07 .38

• Individual Support Areas studied did not show statistically significant differences in student performance

• ASL: fingerspelling, use of space, plurality, equations, introducing an item

• Audio: graphs, equations, drag and drop, images, decimals

• Implications for Guidelines – purposefully use FS, equations, and incorporate graph/image description as linguistically appropriate


Cognitive Lab Research Areas

• Grades 3 - 5

• Sign: use of space, perspective, equations, fingerspelling, plurality, numerals in answers

• Audio: graphics and images, drag and drop, expressions/equations, numerals in answers

• Grades 6 – 8 and Grades 9 – 12

• Sign: graphics and images, expressions/equations, fingerspelling, diamond (item set-up)

• Audio: graphics and images, drag and drop, expressions/equations, numerals, decimals

Cognitive Lab Research Methodology

• Students were asked to complete 4-5 item pairs using a computer based testing system with embedded audio or sign support.

• Each item pair focused on one audio or sign representation issue.

• Students were asked to explain any difficulties they had answering the item, any parts of the audio/sign representation they found confusing or helpful, and were asked to report which audio/sign representation they preferred

Representation A: DetailedItem stem: This graph shows the number of people who used the Internet each year in the United States from nineteen ninety seven to two thousand three. Graph description: Line graph title, U.S. Internet Use. The horizontal label is Year. The vertical label is People, in millions. This data is approximate. For each point on the graph, the year is followed by the number of people in millions. Nineteen ninety seven, fifty. Nineteen ninety eight, seventy eight. Nineteen ninety nine, one hundred two. Two thousand, one hundred twenty two. Two thousand one, one hundred forty two. Two thousand two, one hundred fifty nine. Two thousand three, one hundred sixty one.

Representation B: GeneralItem stem: This graph shows the average temperature in Texas. In January, March, May, July, September, and November. Graph description: The graph title is Average Temperature. The horizontal axis is month. The vertical axis is temperature, in Fahrenheit. The graph shows a line with temperatures noted in January, March, May, July, September, and November.

Student Demographics: Need

Elementary School

Middle School High School

Need Number Percentage

Number Percentage

Number Percentage

Low Vision 3 10% 4 13% 10 29%

Deaf/HH 17 59% 14 45% 18 51%

Reading-based

LD5 17% 10 32% 7 20%

ELL 4 14% 3 10% 0 --

Student Preferences for Audio: Equations

School Level

NeedFirst Item (equation

read)

Second Item

(equation not read)

No Preference

Elementary School

Low Vision3 0 0

100% -- --

Reading-based LD

2 3 040% 60% --

ELL1 1 2

25% 25% 50%

Middle School

Low Vision4 0 0

100% -- --

Reading-based LD

7 1 2

70% 10% 20%

ELL2 1 0

67% 33% --

High School

Low Vision10 0 0

100% -- --

Reading-based LD

6 1 086% 14% --

Student Preferences for Audio: Diagrams and Images

School Level

Need

First Item (detailed

information read)

Second Item

(general description provided)

No Preference

Elementary School

Low Vision2 1 0

67% 33% --

Reading-based LD

2 2 140% 40% 20%

ELL3 1 0

75% 25% --

Middle School

Low Vision0 2 2-- 50% 50%

Reading-based LD

7 2 078% 22% --

ELL2 0 1

67% -- 33%

High School

Low Vision5 1 0

83% 17% --

Reading-based LD

4 2 067% 33% --

Insights from Students: Audio I liked how it actually told me what the

equation was. (317) The read aloud accommodation is helpful

sometimes because sometimes I can’t really focus on what I’m reading. (210)

[equations read aloud] You know better the problem – you can think about it faster if the computer tells you the problem. (112)

[when shapes are described] You don’t have to think that hard. It helped me answer better. (106)

Student Preferences for Sign: Finger Spelling

School LevelFirst Item (ASL sign)

Second Item (ASL

sign + finger

spelling)

No Preference

Elementary School

Number 8 6 1

Percent 53% 40% 7%

School LevelFirst Item (ASL sign)

Second Item (ASL

sign + finger

spelling)

No Preference

Middle School Number 3 8 1

Percent 25% 67% 8%

School LevelFirst Item

(finger spelling)

Second Item (ASL

sign)

Third Item (ASL sign +

finger spelling)

No Preference

High School Number 0 9 8 0

Percent -- 53% 57% --

Student Preferences for Sign: Diamond Presentation

School Level

First Item (introduce item with question)

Second Item (introduce item with

information)

No Preference

Middle School Number 8 5 0

Percent 62% 38% --

High School Number 11 0 2

Percent 85% -- 15%

Insights from Students: Sign

Some of the English words I don’t know so I can watch the sign. (262)

I can understand questions better when they are signed, not finger-spelled. (156)

ASL is our language. English is not bad but ASL is my language. … When you read the English, it is hard to understand. But the signs help. (167)

Measured Progress ©2012 GAAP: Guidelines for Accessible Assessments Project June 26 th, 2014 CCSSO National Conference on Student Assessment New Orleans,

Documents

audio guideline issuesconduct

assessmentprovide guidelines

audio guidelinetext

preprogrammed audio

test administration

paperbased test materials

white background

gaap project background