National Accessible Reading Assessment Projects PARA Project Overview, Results, Next Steps Martha Thurlow and Deborah Dillon.

National Accessible Reading Assessment Projects

PARA Project

Overview, Results, Next Steps

Martha Thurlow and Deborah Dillon


Overview: PARA Partners

Partnership for Accessible Reading Assessment (PARA) consists of:

• University of Minnesota - NCEO (National Center on Educational Outcomes) and Department of Curriculum and Instruction

• CRESST (Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles) and University of California, Davis

• Westat


PARA Research Program Assumptions

• Assumption #1: We don’t know everything about what goes into accessible reading assessment yet.

• Assumption #2: Preliminary research must inform design of “testlets.”

• Assumption #3: Both preliminary and experimental research will inform our knowledge of Principles and Guidelines (Goal 3) for future assessments.


Three Major Research Question Areas

• What characteristics of students require more accessible assessment of reading?

• What characteristics of current assessment practices hinder accessibility?

• What characteristics would accessible assessment have?


PARA Activities and Findings

• Disability Reports• Differential Item Functioning

Analysis• Differential Distractor Functioning

Analysis• Test Characteristics Study• Student Characteristics Pilot

Study


Disability Reports*

Addressing disabilities that seem to affect reading . . . . . some of these are the focus of our projects and some are not

• Visual Impairments

• Deaf or Hard of Hearing

• Autism

• Learning Disabilities

• Mental Retardation

• Speech or Language Impairments

• Emotional or Behavioral Disabilities

* GAC members were instrumental in reviewing/writing these reports.


Differential Item Functioning (DIF) Analysis

• Used to examine differences between students with and without disabilities – to see whether items functioned differently.

• Found that many items exhibited DIF, especially for grade 9 students with disabilities, most often in the second half of the assessment.

• Several issues with the data limited findings – NRT, no access to category of disability or accommodations information; concern about high omission rate.


Differential Distractor Functioning (DDF) Analysis

• Used to examine patterns of incorrect answers to distractors – to see whether items functioned differently.

• Found that many items exhibited DDF for students with disabilities in grade 9, especially in the second half of the assessment.

• Several issues with the data also may limit these findings – NRT, no access to category of disability, accommodations information; concern about high omission rate.


Test Characteristics Study

• Examined test specifications for purposes and constructs, number of items assigned to constructs, and types of items typically found.

• Found that not all standards in states are assessed; yet, more than one-third of states assessed some form of foundational skills; further, in most states, comprehension and analysis/interpretation are considered an extension of foundational skills.

• Not all states posted their test specs/blueprints on Websites; these states are being pursued now to complete the analysis.


Student Characteristics Pilot Study

• Identifying students whose teachers believe have reading skills that are not accurately measured by state reading assessments.

• Found that teachers are able to identify students, but did not identify many with comprehension limitations that obscured other reading skills and those who have strengths outside of what most reading tests cover.

• Issues with follow-up measures made it difficult to verify teachers’ beliefs about skills.


PARA’s Next Steps

• Chunking Study• Student Characteristics Study• Motivation Study


Chunking Study


Purpose of Study

To examine the effects of chunking reading passages on the performance of students with disabilities, and to compare this effect to the effect on the performance of students without disabilities.


Chunking refers to the concept of building breaks directly into the test booklet by separating the passage into more manageable “chunks,” and adding relevant test questions directly beneath the chunk.

Definition


Participants: Grade 8 students with and without disabilities

Passages: Three passages with multiple choice items. All were released state assessment items, for which permission to use was obtained

Participants and Passages


Students with and without disabilities will be randomly assigned to either Version A (standard version) or Version B (chunked version). All students will be given background questions, feedback question related to fatigue and mood, and a student motivation scale.

Design


Student Characteristics Study


Purpose of Study

To identify students whose teachers believe have reading skills that are not accurately measured by state reading assessments, and follow-up with students.

To check on the prevalence of “less accurately measured students” (LAMS) with various characteristics.


Types of Students

1. Fluency limitations obscure comprehension skills

2. Comprehension limitations obscure other reading skills

3. Strengths exist outside of what most reading tests cover

4. Responds poorly to standardized testing circumstances or materials


Research Questions

• How well can teachers identify LAMS?

• How well can teachers document their rationale?

• How well can a brief examination confirm or disconfirm teacher judgments?


Procedures

• Teachers complete nomination questionnaire

• Researchers interview teachers

• Researchers interview students

• Researchers examine students

Students are 4th and 8th grade students with and without disabilities


Examination Protocols

1. CBM, listen to text with immediate retelling

2. CBM, read text with immediate retelling

3. CBM, choice to read or listen to text, and do standard multiple choice questions

4. CBM, think aloud

After interview, student examined with:


Prevalence Study

Teachers complete survey about all of their LAMS, indicating which of the types they have in their classroom and their characteristics


Designing Large Scale Reading Assessments that

Increase Students’ Motivation during the

Assessment


To examine whether improving the motivational characteristics of a large-scale reading assessment increases its accessibility for students with disabilities, and in so doing provides a more valid assessment of these students’ reading proficiency due to their increased engagement.

Purpose of Study


Research Questions

1. Does the option of exerting choice in the selection of passages on a large-scale reading assessment improve the accessibility of the assessment and lead to increased comprehension due to improved motivation and engagement for students with disabilities and general education students?

2.When students with disabilities and general education students are given the option to choose literary and informational passages that are included on large-scale reading assessments, do they perform better on one text type or the other?


3. Are students who are identified as being more motivated to read on general motivation to read measures (e.g., scales and questionnaires) more proficient in reading when given a choice of reading passages on large-scale assessments?

4. Are students who are identified as being more situationally motivated to read, as reported on questions following each passage, more proficient in reading those passages?

Research Questions – cont.


• Participants: Students will be selected from grades 4 and 8. Students with disabilities as well as general education students will be in the experimental condition; comparable students will be in the control condition.

• Setting: Computer lab/classroom, untimed testing situation, appropriate accommodations for students with various disabilities.

Participants & Setting


DesignIn the counterbalanced design of the study we are looking to see if there is increased performance between students (both general education and those with disabilities) who are given choice (C) in selecting the passages they read and those who are not (NC). A stratified random assignment of students will be necessary to ensure that students representing particular disabilities are randomly assigned to the experimental and control condition. Students will take the version of the test they are assigned.


Design—cont.

• Students in the experimental conditions will be allowed to “design your own assessment.”

• The assessments will include 4 total passages (two literacy-fiction and two informational-expository—

based on the NAEP 2009 Reading Framework). • Separate pools of passages will be provided for

4th and 8th grades to select from; each passage will be followed by 5-6 multiple choice items.


“Choice Condition”

• Students in the “choice” condition will be presented with short (1-2 sentence) descriptions/summaries for 6 literary-fiction passages. Students choose 2 of the 6 passages to include in their individually designed assessment.

• Students will be presented with short (1-2 sentence) descriptions or summaries for 6 informational-expository passages. Students choose 2 of these 6 passages to include in their individually designed assessment. Students will be given time to read the descriptions prior to the initiation of the test.

• After selecting the 4 total passages, students will be asked to briefly state why they chose these passages.


“Control Condition”

• “Control” condition students will read 2 literacy-fiction and 2 informational-expository passages from the same high interest pool of passages, but they will not be allowed to choose the particular passages that will be included in their assessment.

• “Control” condition students will be randomly assigned passages (Note. These passages will have been ranked as low interest by 4th and 8th grade students during a pre-calibration study).


“Control Condition”

• “Control condition” students will follow the same procedures as the experimental conditions students: Students will be presented with short (1-2 sentence) descriptions or summaries for 6 literary-fiction passages and 6 informational-expository passages. Students will be asked to rate these passages according to level of interest.


Passage Selection Criteria

• Passages used in this assessment will be selected from existing literature and textbooks based on broad appeal to students at the 4th and 8th grades, respectively. There will be separate pools of passages for 4th and 8th grades.

• Children’s literature scholars and the research literature will be consulted to select high interest passages.


Passage Selection Criteria—cont.

• Reading passages will be chosen to represent two text types and one specific form of text within the text types: informational—exposition; & literary—fiction.

• 18-24 high interest passages at the 4th grade level and the same number at the 8th grade level will be secured; half of the passages will be literary and half will be informational texts.

• Short (1-2 sentence summaries will be written for each passage).


Passage Selection Criteria—cont.

• To determine the difficulty of the passages for 4th and 8th grade readers, each passage will be rated as equivalent on factors such as difficulty and accessibility (using multiple reading ability measures) and structure (e.g., text grammar or top-level structure).

• Permissions will be procured for passages; more passages will be secured than needed on the actual test and more items for each passage will be written (e.g., due to the calibration process and removal of weak items).


Skills Assessed: The Use of Cognitive Targets

• Item writers will use the 2009 NAEP cognitive targets to create 10 items to follow each of the 18-24 passages for each grade level.

• The tasks or questions students answer after reading passages will match the content and key ideas of individual passages rather than being generic questions.


Attending to Issues of Motivation• Texts and items will be ordered from easy to difficult to

foster motivation via improved self efficacy. • Situated motivation questions will be woven into the test

booklets for the choice condition only and will be placed after the items that go with each question; these questions will indicate students’ perceptions of the texts they read (e.g., difficulty; interest) and their sense of self-efficacy in reading and completing the items following the passage (the task).

• A pre-assessment general motivation survey will be given to all students to provide information on student feelings about “self as reader” (e.g., Motivation for Reading Questionnaire-MRQ).


• Demographic data on all students will be collected, including whether they are students with a disability and/or students who are English language learners.

• Observational data and verbal protocols will be obtained from students to determine their perspectives on assessment instruments, tasks, and contexts.

National Accessible Reading Assessment Projects PARA Project Overview, Results, Next Steps Martha Thurlow and Deborah Dillon.

Documents

reading skills

state assessment items

reading tests cover

characteristics of students

definition slide

nceo national center

disabilities passages

deborah dillon slide