Florida Alternate Assessment Technical Report 2007 … · Florida Alternate Assessment Technical Report 2007-2008 . Prepared by Measured Progress for the Florida Department of Education

Florida Alternate Assessment

Technical Report 2007-2008

Prepared by Measured Progress for the Florida Department of Education

TABLE OF CONTENTS SECTION I: OVERVIEW, BACKGROUND, AND KEY COMPONENTS OF THE VALIDITY EVALUATION1

CHAPTER 1. VALIDITY STATEMENT..............................................................................................................................1 CHAPTER 2. OVERVIEW OF THE FLORIDA ALTERNATE ASSESSMENT ...........................................................................3

2.1 Florida’s Assessment Context.......................................................................................................................4 2.1.1 Core Beliefs............................................................................................................................................................... 5 2.1.2 Stakeholders .............................................................................................................................................................. 5

2.2 Purpose of the Florida Alternate Assessment ...............................................................................................6 2.3 Uses of the Florida Alternate Assessment.....................................................................................................6 2.4 Florida Alternate Assessment Participation .................................................................................................7 2.5 Florida Alternate Assessment Content..........................................................................................................8

2.5.1 Standards Review Institute, July 2005 ...................................................................................................................... 8 2.5.2 Reading and Language Arts Access Points ............................................................................................................... 9 2.5.3 English Language Arts Content ................................................................................................................................ 9 2.5.4 Mathematics Content .............................................................................................................................................. 10 2.5.5 Science Content....................................................................................................................................................... 10

2.6 Florida Alternate Assessment Alignment and Linkages..............................................................................10

SECTION II: TEST DEVELOPMENT, ADMINISTRATION, SCORING, AND REPORTING ...........................11

CHAPTER 3. TEST DEVELOPMENT ..............................................................................................................................11 3.1 Item-Design of the Florida Alternate Assessment.......................................................................................12 3.2 Item Components ........................................................................................................................................15 3.3 Florida Alternate Assessment Field Test: Blueprints .................................................................................15 3.4 Florida Alternate Assessment Field Test: Item Development .....................................................................24 3.5 Florida Alternate Assessment Field Test: Bias and Sensitivity Review ......................................................25 3.6 Florida Alternate Assessment Field Test: Content Review.........................................................................26 3.7 Florida Alternate Assessment Field Test: Administration ..........................................................................27 3.8 Florida Alternate Assessment Field Test: Statistics....................................................................................28 3.9 Florida Alternate Assessment Field Test: Survey Results...........................................................................29 3.10 Florida Alternate Assessment: Revisions Based on Field Test Results.......................................................30

CHAPTER 4. TRAINING AND ADMINISTRATION ...........................................................................................................33 4.1 Administrator Training ...............................................................................................................................33

4.1.1 Professional Development....................................................................................................................................... 33 4.1.2 Administration Manual ........................................................................................................................................... 33 4.1.3 Training DVD ......................................................................................................................................................... 34

4.2 Operational Test Administration ................................................................................................................35 4.2.1 Operational Test Survey Results ............................................................................................................................. 36

CHAPTER 5. SCORING .................................................................................................................................................39 5.1 Decision Rules for Scoring .........................................................................................................................39 5.2 Scoring Rubric ............................................................................................................................................41 5.3 Scoring Process ..........................................................................................................................................43

5.3.1 Handling of Incoming Forms .................................................................................................................................. 43 CHAPTER 6. SCANNING...............................................................................................................................................45

6.1 Data Security ..............................................................................................................................................49 6.2 Electronic Records......................................................................................................................................49 6.3 Physical Records ........................................................................................................................................50 6.4 Data Disposal .............................................................................................................................................50 6.5 Secure Test Material Distribution & Return...............................................................................................51

CHAPTER 7. REPORTING .............................................................................................................................................53 7.1 Report Shells...............................................................................................................................................53 7.2 Decision Rules for Reporting......................................................................................................................54

SECTION III—TECHNICAL CHARACTERISTICS OF THE FLORIDA ALTERNATE ASSESSMENT.........55

CHAPTER 8. TECHNICAL CHARACTERISTICS OF THE FLORIDA ALTERNATE ASSESSMENT ..........................................55 8.1 Performance Standards and Student Results ..............................................................................................55 8.2 Item Statistics..............................................................................................................................................62

8.2.1 Difficulty Indices .................................................................................................................................................... 63 8.2.2 Item Discrimination ................................................................................................................................................ 64 8.2.3 Summary of Item Analysis Results ......................................................................................................................... 65

8.3 Differential Item Functioning .....................................................................................................................66 CHAPTER 9. RELIABILITY ...........................................................................................................................................73

i

9.1 Reliability of Performance Level Categorization........................................................................................79 9.1.1 Computing Accuracy and Consistency ................................................................................................................... 80 9.1.2 Accuracy and Consistency Results.......................................................................................................................... 82

SECTION IV: THE VALIDITY EVALUATION .........................................................................................................85

CHAPTER 10. REVISITING THE VALIDITY EVALUATION QUESTIONS ........................................................................85

REFERENCES .................................................................................................................................................................87

APPENDICES...................................................................................................................................................................91

APPENDIX A–FLORIDA STAKEHOLDER LISTS ..............................................................................................................93 APPENDIX B–SAMPLE ITEM: FIELD TEST FORMAT ....................................................................................................105 APPENDIX C–SURVEYS AND RESULTS ........................................................................................................................113 APPENDIX D–SAMPLE ITEM: OPERATIONAL TEST FORMAT.......................................................................................129 APPENDIX E–REPORT SHELLS ....................................................................................................................................133 APPENDIX F–PARENT AND TEACHER BROCHURES .....................................................................................................147 APPENDIX G–ITEM SPECIFICATIONS DOCUMENT........................................................................................................155 APPENDIX H–RAW SCORE CUMULATIVE DISTRIBUTIONS ..........................................................................................181 APPENDIX I–ITEM ANALYSIS RESULTS.......................................................................................................................193 APPENDIX J–DECISION ACCURACY AND CONSISTENCY .............................................................................................227

ii

SECTION I: OVERVIEW, BACKGROUND, AND KEY COMPONENTS OF THE VALIDITY

EVALUATION Chapter 1. VALIDITY STATEMENT

This report describes several technical aspects of the Florida Alternate Assessment in an

effort to contribute to the accumulation of validity evidence to support Florida Alternate Assessment

score interpretations. Because it is the interpretations of test scores that are evaluated for validity, not

the test itself, this report presents documentation to substantiate intended interpretations (AERA,

1999). Each of the sections in this report contributes important information to the validity argument

by addressing one or more of the following aspects of the Florida Alternate Assessment: test

development, test alignment, test administration, scoring, reliability, achievement levels and

reporting. The report further outlines future plans of the Florida Department of Education (FLDOE)

to investigate other technical validity and reliability aspects of the assessment system.

The Florida Alternate Assessment outlined in this report is based on, and aligned to, the

Sunshine State Standards Access Points in reading, writing and mathematics. The science

assessments are aligned to the Sunshine State Standards due to the fact that at the time of

development the Access Points for science had not yet been created. Intended inferences from the

Florida Alternate Assessment results are about student achievement on Florida‘s reading, writing,

mathematics and science content standards and these alternate achievement inferences are meant to

be useful for program and instructional improvement and as a component of school accountability.

The Standards for Educational and Psychological Testing (1999) provides a framework for

describing sources of evidence that should be considered when constructing a validity argument.

These sources include evidence based on the following five general areas: test content, response

processes, internal structure, relationship to other variables, and consequences of testing. Although

Chapter 1—Validity Statement 1 2007–08 Florida Alt Technical Report

each of these sources may speak to a different aspect of validity, they are not distinct types of

validity. Instead, each contributes to a body of evidence about the comprehensive validity of score

interpretations.

Chapter 1—Validity Statement 2 2007–08 Florida Alt Technical Report

Chapter 2. OVERVIEW OF THE FLORIDA ALTERNATE ASSESSMENT

The Individuals with Disabilities Education Act (IDEA) requires that students with

disabilities be included in each state‘s system of accountability and that students with disabilities

have access to the general curriculum. The No Child Left Behind Act (NCLB) also speaks to the

inclusion of all students in a state‘s accountability system by requiring states to report student

achievement for all students as well as for specific groups of students (e.g., students with disabilities,

students for whom English is a second language) on a disaggregated basis. These federal laws reflect

an ongoing concern about equity. All students should be academically challenged and taught to high

standards. The involvement of all students in the educational accountability system provides a means

of measuring progress toward that goal.

To provide an option for participation of all students in the state‘s accountability system,

including those for whom participation in the general statewide assessment (the Florida

Comprehensive Assessment Test [FCAT]) is not appropriate, even with accommodations, Florida

has developed the Florida Alternate Assessment. The alternate assessment design for Florida is

based on the Sunshine State Standards Access Points for Students with Significant Cognitive

Disabilities in Language Arts (Reading and Writing), Mathematics, and Science. Access Points

represent the essence of the Sunshine State Standards with reduced levels of complexity:

Participatory, Supported, and Independent, with the Participatory Level being the least complex. The

Florida Alternate Assessment was developed to allow students an opportunity to progress through all

three levels of complexity per item. This tiered process provides students the opportunity to work to

their potential for each item in each content area. This is critical as educators seek to provide access

to the general education curriculum and foster higher expectations for the wide diversity of students

with significant cognitive disabilities. It is expected that only students with the most significant

cognitive disabilities who are eligible under IDEA will participate in the Florida Alternate

Assessment.

Chapter 2—Overview of the FLALT Assessment 3 2007–08 Florida Alt Technical Report

2.1 Florida’s Assessment Context

Florida‘s focus on educational accountability began in 1991 with the school improvement

and accountability legislation. The intent of this legislation was to ensure higher levels of

achievement for all students and more accountability for schools. In 1996, the Sunshine State

Standards were adopted by the State Board of Education and the Florida Comprehensive Assessment

Test (FCAT) was authorized by the legislature. During this same time period, efforts were made to

build capacity within school districts to develop and implement local alternate assessment tools for

students who could not participate in FCAT. In 1999, the legislature passed the A+ Plan for

Education which increased standards and accountability for students, schools, and educators. The

assessment system included reading and math in grades 3 through 10, writing in grades 4, 8 and 10,

science in grades 5, 8, and 11, and the development of a system for calculating individual academic

growth over a year‘s time. Florida‘s school grading system was implemented in 1999.

In 2005, Florida began the process of revising the Sunshine State Standards. As a part of this

revision, access points for students with significant cognitive disabilities were developed. These

access points represent the core intent of the standards with reduced levels of complexity. In

alignment with this activity, in 2007, Florida began design and development of a statewide alternate

assessment based on alternate achievement standards. The intent was to replace the system of local

assessments and state reporting aligned to previous standards with a new statewide assessment

aligned to the newly adopted access points.

Currently, Florida provides two statewide assessments, the general assessment (FCAT) and

an alternate assessment based on alternate achievement standards (Florida Alternate Assessment).

Reading and mathematics are assessed in grades 3 through 10. Writing assessments are provided for

grades 4, 8, and 10 and science assessments occur in grades 5, 8, and 11.


2.1.1 Core Beliefs

The mission of the Florida Department of Education is to lead and support schools and

communities in ensuring that all students achieve at the high levels needed to lead fulfilling and

productive lives, to compete in academic and employment settings, and to contribute to society. The

core beliefs that are held are that:

° all students can learn,

° all students should have access to general curriculum,

° all students should be challenged, and

° all students should have opportunities to demonstrate what they know and can do

2.1.2 Stakeholders

There have been many stakeholders involved with the development and implementation of

the Florida Alternate Assessment. An advisory committee, representing the perspectives of teachers,

and administrators, provided input during the development of the assessment. In addition, teacher

work groups were formed at several points in the development process. Starting in July 2005 the

Access Point work was accomplished by staff from Accountability and Assessment for Students

with Disabilities Project (PAEC) and the Accommodations and Modifications for Students with

Disabilities Project (FSU); in addition teachers and university personnel with special education and

content expertise were involved in the process. A bias and sensitivity work group, composed of

general and special education teachers and specialists, was formed. These educators reviewed the

initial passages for the reading assessments prior to the writing of items and reviewed a sampling of

science items. Another group of teachers worked to review all of the items for content, alignment to

the Access Points and appropriateness for the population of students being assessed. Another group

of special education teachers participated in the field testing, providing valuable feedback about the

test design. (Stakeholder lists can be found in Appendix A.)


2.2 Purpose of the Florida Alternate Assessment

Consistent with the state‘s general assessment (FCAT), the purposes of the Florida Alternate

Assessment are as follows: (1) assess the annual learning gains of each student toward achieving the

Sunshine State Standards appropriate for the student‘s grade level; (2) provide data for making

decisions regarding school accountability and recognition; (3) assess how well educational goals and

curricular standards are met at the school, district, and state levels; (4) provide information to aid in

the evaluation and development of educational programs and policies; and (5) provide information

on the performance of Florida students compared with that of other students across the United States.

2.3 Uses of the Florida Alternate Assessment

Florida Alternate Assessment results are provided at the student, school, district, and state

levels. Interpretative brochures for parents and teachers are sent to schools with the Florida Alternate

Assessment Student Score Reports. Educators, parents, and students are encouraged to use the

reported scores to inform instruction and chart student progress in meeting the Access Points.

Results of the Florida Alternate Assessment show educators how students with significant

cognitive disabilities are progressing toward learning the knowledge and skills contained in the

Sunshine State Standards Access Points. They can be used to assist IEP teams in developing annual

goals and objectives. The IEP team should examine the results in conjunction with other

informationœsuch as progress reports, report cards, and parent and teacher observationsœto see what

additional instruction is needed and in what areas.

The results can also be used to improve instructional planning. For example, a student

exhibiting skills within the participatory level of complexity at an advanced performance level may

be ready for work that is more difficult and will most likely be instructed on a combination of

Access Points at both the participatory and supported levels. Students‘ scores may also indicate a


need to have the curriculum adjusted or for students to be provided with additional supports and

learning opportunities.

2.4 Florida Alternate Assessment Participation

The Florida Alternate Assessment is an alternate achievement standards-based assessment

designed specifically for students with significant cognitive disabilities. There are three assessment

options; participating in the FCAT, participating in the FCAT with accommodations or participating

in the Florida Alternate Assessment. Students who meet the criteria to participate in the Florida

Alternate Assessment are unable to participate in the FCAT even with accommodations and are

working on alternate content standards that are measured against alternate achievement standards.

Individual Educational Plan (IEP) teams are responsible for determining whether students with

disabilities will participate in alternate assessment. The IEP team should consider the student‘s

present level of educational performance in reference to the Sunshine State Standards. In order to

facilitate informed and equitable decision making, IEP teams should answer each of the following

questions when determining whether or not a student should participate in the Florida Alternate

Assessment:

Table 2-1. 2007-08 Florida Alternate Assessment: Student Participation Criteria Questions to Guide the Decision-Making Process to Determine YES NOWhether a Student Takes FCAT or the Florida Alternate Assessment

1. Is the student unable to master the Sunshine State Standards even with appropriate and allowable course accommodations?

–– ––

2. Is the student‘s demonstrated cognitive ability the primary reason for the inability to master these standards? –– –– 3. Is the student participating in a modified curriculum based on competencies in the Sunshine State Standards Access Points for all academic areas? –– –– 4. Does the student require extensive direct instruction in academic and vocational competencies as well as domestic, community living and leisure activities? –– ––

5. Does the student have deficits in adaptive behavior, as demonstrated by the inability to function effectively and independently in everyday living skills (interpersonal and social interactions) across a variety of settings?

–– ––


If the IEP team determines that all five of the questions accurately characterize a student‘s

current educational situation, then the Florida Alternate Assessment should be used to provide a

meaningful evaluation of the student‘s current academic achievement. If —yes“ is not indicated in all

five areas, then the student should participate in the Florida Comprehensive Assessment Test

(FCAT) with or without accommodations. Figure 2-1 shows 2007-08 participation rates.

Figure 2-1. 2007-08 Florida Alternate Assessment: Participants by Grade Level

2388 2478 2519

2971 3078 3033

2280 2310

2234

0

500

1000

1500

2000

2500

3000

3500

03 04 05 06 07 08 09 10 11 Grade-Level

Num

ber o

f Stu

dent

s

2.5 Florida Alternate Assessment Content

2.5.1 Standards Review Institute, July 2005

In 2005, the Bureau of Exceptional Education and Student Services initiated the development

of extensions (curriculum expectations) for students with significant cognitive disabilities for the

Sunshine State Standards in Language Arts, Mathematics, and Science. To begin this process, school

districts were invited to nominate participants from across the state representing exceptional student

education, general education, English-language learners, and parents to write draft extensions for

three levels of complexity: Independent, Supported, and Participatory. The draft extensions were

aligned to the 1996 Sunshine State Standards benchmarks. In December 2005, the extensions for


Language Arts and Mathematics were posted for public review in an online survey. A total of 164

responded to the language arts survey and 42 responded to the mathematics survey.

2.5.2 Reading and Language Arts Access Points

Beginning in January 2006, staff from the Accountability and Assessment for Students with

Disabilities Project (PAEC) and the Accommodations and Modifications for Students with

Disabilities Project (FSU) worked together to align the draft extensions to the revised benchmarks of

the Sunshine State Standards for Language Arts. Throughout the process, teachers and university

personnel with expertise in language arts and those with expertise in curriculum for students with

disabilities were consulted, although no formal writing team was established. The extensions were

renamed —access points.“ In April 2006, the access points were included in an online survey with the

revisions to the Reading and Language Arts Sunshine State Standards. Access points were aligned

with further revisions to the general education standards. The final draft was adopted by the State

Board of Education on January 25, 2007.

Students in the Florida Alternate Assessment are assessed utilizing the Access Points from

the Sunshine State Standards in the areas of reading, writing, and mathematics. Access Points

represent the essence of the Sunshine State Standards benchmarks at reduced levels of complexity.

Science Access Points were not written in time for the science field test or 2007-08 operational test.

Instead, the Sunshine State Standards were expanded by the science developer for these

administrations. Future administrations of the Florida Alternate Assessment in science will be based

on Access Points.

2.5.3 English Language Arts Content

For English Language Arts, three reading strands are assessed: Reading Process, Literary

Analysis, and, at the higher grades, Information and Media Literacy. Two writing strands are

assessed: Writing Process and Writing Application.


2.5.4 Mathematics Content

Mathematics content is broken into Big Ideas and Supporting Ideas for grades 3 through 8.

There are three Big Ideas at each grade level and four Supporting Ideas that cover Algebra,

Geometry and Measurement, Number and Operations, and Data Analysis. In grades 9 and 10,

content is structured in terms of six Secondary Bodies of Knowledge: Algebra, Discrete

Mathematics, Geometry, Probability, Statistics, and Financial Literacy.

2.5.5 Science Content

The science content assessed for the 2007-08 Florida Alternate Assessment included three

major content areas–Physical and Chemical Sciences, Earth and Space Sciences, and Life and

Environmental Sciences–and the process of Scientific Thinking.

2.6 Florida Alternate Assessment Alignment and Linkages

The Florida Department of Education has contracted with Claudia Flowers and Shawnee

Wakeman from the University of North Carolina at Charlotte to conduct an alignment study of the

Florida Alternate Assessment and Sunshine State Standards Access Points. The criteria used for the

alignment study were developed by the National Alternate Assessment Center (NAAC). The

alignment methodology utilizes eight alignment criteria such as, the academic nature of the content,

the fidelity of the content to the original grade level standards and the accessibility of the

assessment. The full Florida Alternate Assessment Alignment Report is available through the Florida

Department of Education.


SECTION II: TEST DEVELOPMENT, ADMINISTRATION, SCORING, AND

REPORTING Chapter 3. TEST DEVELOPMENT

In April of 2007 the Florida Department of Education entered into a development contract

with Measured Progress. The new Florida Alternate Assessment was developed in response to a

Request for Proposal (RFP) disseminated by the Florida Department of Education requesting a new

design for their alternate assessment that would be based on their newly developed Sunshine State

Standards with Access Points. The department wanted a new assessment that would include multiple

item types and assessment levels within a primarily performance task type of assessment. This new

design further needed to allow students a tiered entry into the assessment for students working at the

varying levels of complexity.

Prior to starting the development process a Test Designs, Blueprints and Item Specifications

for Reading, Writing, Mathematics and Science document was developed and was presented to the

Florida Department of Education and Florida Alternate Assessment Advisory in April of 2007 at an

Alternate Assessment Advisory meeting. The initial design that was presented at this meeting did not

include the scaffolding at the Participatory level that is outlined in the design section that follows.

This change in the initial design came about as a result of the concerns that the advisory members

had for the students working within the lowest level of complexity. They felt that an item presented

only one time with a possibility of being either right or wrong did not give these students the

opportunity to show what they know and are able to do. The advisory members were also presented

with the blueprints and asked for their input. A few changes were made as an outcome of their input,

for example, in reading the concept of Comparing and Contrasting was removed from grade 3 and in

mathematics for grades 9 and 10 Financial Literacy was added to the assessment blueprint. The

Chapter 3—Test Development 11 2007–08 Florida Alt Technical Report

document was finalized and the development that occurred after this point referenced this document

for design, blueprints, and item specifications. The discussion on the design and blueprints below is

based upon this final document and reflect the changes that the advisory committee recommended.

This final design was then presented at the Florida Special Education Institute in July of 2007

in front of approximately 500 educators. The design was well received and no further adjustments

were made to the overall design at that point in time.

3.1 Item-Design of the Florida Alternate Assessment

Designed specifically for students with significant cognitive disabilities, the Florida Alternate

Assessment is a performance-based test that is aligned with the Sunshine State Standards Access

Points for Language Arts (Reading and Writing) and Mathematics. Science items are based on a

balance of the content area of the current Sunshine State Standards and Strand B, Standard 4 of the

Sunshine State Standards for Special Diploma. The assessment measures student performance based

on alternate achievement standards.

The Sunshine State Standards with Access Points consist of the general education strands,

standards and benchmarks, beneath which three levels of skills are linked. These three skill levels

are the Access Points and are referred to as levels of complexity. The three levels of complexity are

participatory, supported, and independent, with the participatory level representing the least complex

skill set and the independent level the most complex. Items were written to address all three levels of

complexity/access points.

Students receive a final score for an item set based on the level at which they answer

correctly. A student starts at the participatory level of complexity within an item set. A student

completing the participatory level item accurately, without assistance, moves on to the supported

level item. If the student is able to complete the supported item, the student is administered the

independent level item. In other words, a student moves up through the Access Point skills as long as


he or she is able to respond accurately and independently and receives a score consistent with the

highest correct response.

Scaffolding is given only at the participatory level, to a student who is unable to complete a

participatory level item accurately and independently and receives a score consistent with highest

correct response. The student is presented the item again with one distracter removed. If the student

is able to accurately respond he/she is given a score of 2. If the student is unable to accurately

respond, he/she is presented the item again with another distracter removed (leaving only the correct

answer) and the student is asked to actively engage with the correct answer and a score of 1 is

recorded. At any point within the participatory level item, if the student will not engage or actively

refuses, the student scores 0 points.

In summary, Florida Alternate Assessment grade-content tests can be thought of as 16-item

tests if the Participatory, Supported, and Independent items are considered in sets. The scoring rubric

does just that, and treats each set as a polytomous item with six possible item scores: 0, 1, 2, 3, 6, 9.

The maximum possible total raw score is 144.

A visual depiction of this process is provided on the next page.


Figure 3-1. 2007-08 Florida Alternate: Item Design Process


3.2 Item Components

Each item set includes an overview, the access points to be assessed, and the materials

needed. The components for each item set are:

Materials Access Point Teacher Will Student Will Scoring

° The Materials column outlines for the educator which materials are needed for the item.

Materials that are provided for the teacher and materials the educator may need to gather

from the classroom are identified. Graphics are named for teachers to use standardized

terminology as needed. The materials generally consist of picture cards, word/picture

cards, word cards, picture/sentence strips, sentence strips, number cards and equation

strips.

° The Access Point column lists the access point that the item is targeting.

° The Teacher Will column consists of a clear set of directions for setting up the item and

scripting for what the teacher should say to the student.

° The Student Will column indicates the response that the educator needs to look for from

the student, taking into consideration the mode of communication appropriate for each

student.

° The Scoring column provides a space for the educator to mark the score the student

received on the item.

For the field test in the fall of 2007 the materials provided for each of the items, picture

cards, sentence strips, etc. were provided in a separate booklet that required the teacher to cut out

and organize all of the materials prior to administering the assessment to a student. A sample item

reflective of the format of the field test can be found in Appendix B.

3.3 Florida Alternate Assessment Field Test: Blueprints

The original blueprints for the Florida Alternate Assessment consisted of 24 items for each

grade level and content area test. Thirty items were developed for the field test at each grade level

and content area. On the following pages are the blueprint charts for each content area. The charts

contain two numbers in each cell; the first number is the number of items developed for the field test


administration and the second number is the number of items that was utilized for the 2008

operational assessment. This second number in each cell reflects the final blueprint. Note that the

final blueprint consists of 16 items per grade level and content area. The overall reduction in items

from a test design of 24 items to a test design of 16 items in most cases reduced the number of items

at the standard level from 6 items to 4 items. This change in the number of items for the operational

assessment is discussed in section 3.10.


Table 3-1. 2007-08 Florida Alternate: Blueprints—Reading Strand 1 Reading Process: Grade

3 Grade

4 Grade

5 Grade

6 Grade

7 Grade

8 Grade

9 Grade

10 Standard 1: Vocabulary Development -The student uses multiple strategies to develop grade appropriate vocabulary. 7/4 6/4 8/4 6/4 7/4 6/4 6/4

LA.7.1.6.1 - use new vocabulary that is introduced and taught directly …. 2/1 2/1 2/1

LA.7.1.6.3 - use context clues to determine meanings of unfamiliar words …. 5/3 2/2

LA.6.1.6.4 - categorize key vocabulary and identify salient features …. 6/4 LA.4.1.6.5 - relate new vocabulary to familiar words…. 1/1 2/1 3/2 LA.4.1.6.6 - identify —shades of meaning“ in related words (e.g., blaring, loud)….

1/1

1/0 0/0 2/1

LA.3.1.6.7 œ The student will use meaning of familiar base words and affixes (prefixes and suffixes) to determine meanings of unfamiliar complex words, …

3/2 2/0 2/1

LA.3.1.6.8 - The student will use knowledge of antonyms, synonyms, homophones, and homographs to determine meanings of words .… 2/0 2/2 1/1

1/1

LA.3.1.6.10 - The student will determine meanings of unfamiliar words by using a dictionary, thesaurus, and digital tools…. 2/2 2/1

Standard 2: Reading Comprehension - The student uses a variety of strategies to comprehend grade level text. 7/4 8/4 7/4 8/5 7/4 8/4 8/4 14/7

LA.3.1.7.2 - The student will identify the author‘s purpose (e.g., to inform, entertain, or explain) in text and how an author‘s perspective influences text ….

2/1 2/2 2/1 3/2 2/1 3/1 3/1 4/2

LA.3.1.7.3 - determine explicit ideas and information in grade-level text, including but not limited to main idea, relevant supporting details, strongly implied message and inference, and chronological order of events ….

4/3 3/1 3/2 3/2 2/2 2/1 3/2 5/2

LA.3.1.7.5 - The student will identify the text structure an author uses (e.g., comparison/contrast, cause/effect, and sequence of events) and explain how it impacts meaning in text….

1/0 2/1 1/0 2/1 2/0 2/1

LA.3.1.7.7 - The student will compare and contrast topics, settings, characters, and problems in two texts …. 1/0 1/1 1/1 1/1 2/1 5/3

(cont‘d)


Strand 2 Literary Analysis: Grade 3

Grade 4

Grade 5

Grade 6

Grade 7

Grade 8

Grade 9

Grade 10

Standard 1: Fiction œ The student identifies, analyzes, and applies knowledge of the elements of a variety of fiction and literary texts to develop a thoughtful response to a literary selection.

8/4 8/4 7/4 8/4 8/4 8/3 10/4 7/3

LA.4.2.1.2 - identify and explain the elements of plot structure, including exposition, setting, character development, problem/resolution, and theme in a variety of fiction;

2/2 4/2 8/4 8/4 8/3

LA.910.2.1.5- describe, discuss, and analyze an author‘s use of literary elements (i.e., theme, point of view, characterization, setting, plot), and explain and analyze different elements of figurative language (i.e., simile, metaphor, personification, hyperbole, symbolism, allusion, imagery) in multiple literary selections…

10/4 7/3

LA.3.2.1.6 - The student will write a book report or review that identifies the main idea, character(s), setting, sequence of events, and problem/solution;

6/2 4/2 7/4

Standard 2: Non-Fiction- The student identifies, analyzes, and applies knowledge of the elements of a variety of non-fiction, informational, and expository tests to demonstrate an understanding of the information presented.

8/4 8/4 8/4 8/3 8/4 8/5 6/4 7/4

LA.3.2.2.2 - The student will use information from the text to answer questions related to explicitly state main ideas or relevant details …. 2/2 4/3 4/2 4/3 4/3 5/3 7/4

LA.3.2.2.3 - The student will organize information to show an understanding of main ideas within a text through charting, mapping, or summarizing ….

6/2 4/1 8/4 4/1 4/1 4/2 1/1

Strand 3 Grades 9œ10: Information and Media Literacy Grade 3

Grade 4

Grade 5

Grade 6

Grade 7

Grade 8

Grade 9

Grade 10

Standard: 910.6 The student uses a systematic process for the collection, processing and presentation of information. 2/2

LA.910.6.2.2 - organize, synthesize analyze and evaluate the validity and reliability of information from multiple sources (including primary and secondary sources) to draw conclusions using a variety of techniques, and correctly use standardized citations;

2/2

LA.910.6.2.3 - write an informational report that integrates information and makes distinctions between the relative value and significance of specific data, facts, and ideas;


Table 3-2. 2007-08 Florida Alternate: Blueprints—Writing Strand 1 Writing Process: Grade 4 Grade 8 Grade

10 Standard 1: 8.3.1 Pre-Writing œ The student will use prewriting strategies to generate ideas and formulate a plan. LA.910.3.1.1 - generating ideas from multiple sources (e.g., brainstorming, notes, journals, discussion, research materials or other reliable sources) based upon teacher-directed topics and personal interests …. LA.8.3.1.2 - making a plan for writing that addresses purpose, audience, main idea, logical sequence, and time frame for completion…. Standard 2: 4.3.2 Drafting œ The student will write a draft appropriate to the topic, audience, and purpose. 9/5

LA.4.3.2.1 - using a pre-writing plan to focus on the main idea with ample development of supporting details that shows an understanding of facts and/or opinions ….

5/2

LA.910.3.2.2 - establishing a logical organizational pattern with supporting details that are substantial, specific, and relevant …. LA.4.3.2.3 - creating interesting leads through the use of quotations, questions, or descriptions …. 4/3

Standard 3: 8.3.3 Revising -The student will revise and refine the draft for clarity and effectiveness. 10/5 10/6

LA.8.3.3.1 - evaluating the draft for development of ideas and content, logical organization, voice, point of view, word choice, and sentence variation ….

6/3 4/3

LA.8.3.3.2 - creating clarity and logic by maintaining central theme, idea, or unifying point and developing relationships among ideas …. 4/2 2/1

LA.910.3.3.4 - applying appropriate tools or strategies to evaluate and refine the draft (e.g., peer review, checklists, rubrics). 4/2

Standard 4: 4.3.4 Editing for Language Conventions - The student will edit and correct the draft for the standard language conventions. 10/5 10/4 8/4

LA.4.3.4.1 - spelling, using spelling rules, orthographic patterns, and generalizations (e.g., r-controlled, diphthong, consonant digraphs, vowel digraphs, silent e, plural for words ending in œy, doubling final consonant, i before e, irregular plurals, CVC words, CCVC words, CVCC words, affixes) and using a dictionary, thesaurus, or other resources as necessary; words used as names (e.g., Uncle Jim, Mom, Dad, Jr.) ….

3/3 2/1 2/1

LA.4.3.4.2 - capitalization for proper nouns, including titles used with someone‘s name, initials…. 2/1 2/0 2/0

LA.4.3.4.3 - punctuation, including end punctuation, apostrophes, commas, colons, quotation marks in dialogue, and apostrophes in singular possessives ….

2/0 2/0 2/2

LA.4.3.4.4 - present and past verb tense, noun-pronoun agreement, noun-verb agreement, subjective and objective pronouns, demonstrative pronouns and conjunctions;

2/1 2/2 2/1

LA.4.3.4.5 - subject/verb and noun/pronoun agreement in simple and compound sentences …. 2/0 2/1

Standard: 4.3.5 Publishing - The student will write a final product for the intended audience. 2/1

LA.4.3.5.1 - prepare writing using technology in a format appropriate to audience and purpose (e.g., manuscript, multimedia)…. 2/1

(cont‘d)


Strand 2 Writing Applications: Grade 4 Grade 8 Grade 10

Standard: 4.4.1 Creative - The student develops and demonstrates creative writing. 8/5 4/3 5/3

L.A. 4.4.1.1 - write narratives based on real or imagined ideas, events, or observations that include characters, setting, plot, sensory details, a logical sequence of events, and a context to enable the reader to imagine the world of the event or experience ….

8/5 4/3 5/3

Standard: 4.4.2 Informational -The student develops and demonstrates technical writing that provides information related to real-world tasks. 6/4 7/3

LA.4.4.2.1 - write in a variety of informational/expository forms (e.g., summaries, procedures, recipes, instructions, graphs/tables, experiments, rubrics, how-to manuals)….

2/2

LA.4.4.2.2 - record information (e.g., observations, notes, lists, charts, map labels, legends) related to a topic, including visual aids as appropriate …. 2/1

LA.4.4.2.3 - write informational/expository essays that contain introductory, body, and concluding paragraphs …. 1/0

LA.910.4.2.4 - write a business letter and/or memo that presents information purposefully and succinctly to meet the needs of the intended audience following a conventional format (e.g., block, modified block, memo, email);

1/1 2/1

LA.910.4.2.5 - write detailed travel directions and design an accompanying graphic using the cardinal and ordinal directions, landmarks, streets and highways, and distances;…

2/1

LA.910.4.2.6 - write a work-related document (e.g., application, resume, meeting minutes, memo, cover letter, letter of application, speaker introduction, letter of recommendation)….

3/1

The grade level number in the benchmark shows the grade level at which the benchmark is first tested; the benchmark may expand at higher grade levels; hence the ellipses. Numbers indicate how many item sets will be written at each grade level for each benchmark.


Table 3-3. 2007-08 Florida Alternate: Blueprints—Mathematics Grade 3 Big Idea Number of

Items IN Level SU Level PA Level

3.1 10/5 At least 1 each access point 4 each access point 4 each access point

3.2 3/2 2 each access point 2 each access point 1 each access point

3.3 9/5 At least 2 each access point

At least 2 each access point


Supporting Ideas

2/1 2/1 2/2 2/0

Alg: 2 G&M: 2 N&O: 1 DA: 1

Alg: 2 G&M: 2 N&O: 1 DA: 1

Alg: 2 G&M: 2 N&O: 1 DA: 1



4.1 8/3

a: 2 b: 1 c: 2 d: 1

a: 1 b: 2 c: 3

3 each access point

4.2 7/4 2 each access point 3 each access point At least 1 each access point

4.3 7/5 At least 1 each

access point, none for a

At least 1 each access point, none

for a


Supporting Ideas

3/2 4/1 1/1

Alg: 2 G&M: 2 N&O: 2

Alg: 2 G&M: 2 N&O: 2

Alg: 2 G&M: 2 N&O: 2



5.1 10/3 4 each access point 4 each access point 4 each access point 5.2 3/2 2 each access point 1 each access point 1 each access point

5.3 9/5 2 each, none for b At least 1 each access point


Supporting Ideas

3/2 3/2 1/1 1/1

Alg: 2 G&M: 2 N&O: 1 DA: 1

Alg: 2 G&M: 2 N&O: 1 DA: 1

Alg: 2 G&M: 2 N&O: 1 DA: 1






6.2 7/3 a: 4 b: 2

a: 4 b: 2 2 each access point




Supporting Ideas 2/1 3/2 3/2

G&M: 2 N&O: 2 DA: 2

G&M: 2 N&O: 2 DA: 2

G&M: 2 N&O: 2 DA: 2




7.1 7/4 a: 4 b: 2

At least 1 each access point 2 each access point

7.2 7/4 2 each access point

a: 2 b: 2 c: 1 d: 1

3 each access point




Supporting Ideas

2/1 3/2 2/1 1/0

G&M: 2 N&O: 2 DA: 1 PR: 1

G&M: 2 N&O: 2 DA: 1 PR: 1

G&M: 2 N&O: 2 DA: 1 PR: 1




a: 3 b: 1 c: 3 d: 1

4 each access point




8.3 3/2 1 each access point 1 each access point 2 at access point

Supporting Ideas 3/1 3/3 3/2

Alg: 2 G&M: 2 N&O: 2

Alg: 2 G&M: 2 N&O: 2

Alg: 2 G&M: 2 N&O: 2

Table 3-9. 2007-08 Florida Alternate: Blueprints—Mathematics Grade 9 Content Area Total number

of items Item Distribution

By Standard

Algebra 3/2 3/2 4/2

A.1: 2 A.2: 2 A.3: 4

Geometry

2/1 3/1 2/0 3/2

G.1: 2 G.2: 2 G.3: 2 G.4: 2

Financial Literacy* 1/0 1/1 3/3

F.1: 1 F.2: 1 F.3: 2

Discrete Math 5/2 2 items to each bullet for each level


Table 3-10. 2007-08 Florida Alternate: Blueprints—Mathematics Grade 10 Content Area Total number

of items Item Distribution

By Standard

Algebra

2/0 2/1 1/1 2/2 3/0

A.4: 2 A.5: 1 A.6: 1 A.7: 2 A.8: 2

Geometry

2/1 3/2 4/1 1/0

G.5: 2 G.6: 2 G.7: 2 G.8: 2

Financial Literacy* 1/1 2/2 2/1

F.1: 1 F.2: 1 F.3: 2

Probability 1/0 1/1

P.1: 1 P.2: 1

Statistics 3/3 S.3: 2 * Grades 9 and 10 are distinguished by complexity of the items

Table 3-11. 2007-08 Florida Alternate: Blueprints—Science Grade 5 Grade 8 Grade 11

Physical and Chemical Sciences 10/6 10/6 11/8 Strand A: The Nature of Matter 5/3 5/2 5/3 Strand B: Energy 3/2 2/2 3/3 Strand C: Force and Motion 2/1 3/2 3/2 Standard 4: The student uses systematic approaches to solve problems* Earth and Space Sciences 8/4 8/2 7/2 Strand D: Process that Shape the Earth 5/3 5/1 4/1 Strand E: Earth and Space 3/1 3/1 3/1 Standard 4: The student uses systematic approaches to solve problems* Life and Environmental Sciences 12/8 12/8 12/6

Strand F: Process of Life 7/3 7/5 7/3 Strand G: How Living Things Interact with Their Environment 5/3 5/3 5/3 Standard 4: The student uses systematic approaches to solve problems* *Items that assess Standard 4 (Scientific Thinking) will also assess a science content benchmark.


3.4 Florida Alternate Assessment Field Test: Item Development

Items were initially developed by Measured Progress Curriculum and Assessment staff. A

lead developer was assigned in each of the content areas and it was their responsibility to oversee all

item development within that content area for the Florida Alternate Assessment. Once an item had

been developed and reviewed by the lead developer the item was then reviewed by a special

education specialist. The curriculum and assessment developer was responsible to make sure that the

item stayed true to the content of the Access Points it was assessing and the special education

specialist reviewed the item for the appropriateness of the topics utilized, materials required and

difficulty of the item for the population of students with significant cognitive disabilities. Items were

also reviewed to ensure that they met the item specifications.

Item specifications were developed and were included in the document Test Designs,

Blueprints and Item Specifications for Reading, Writing, Mathematics and Science which was

approved by the Department in May of 2007. The specifications outlined a variety of item details

such as the length and readabilities of passages for reading, the types of distracters at each of the

levels of complexity, parameters for graphics and the types of topics appropriate for students being

assessed through an alternate assessment. The item specifications document can be found in

Appendix G.

Items were further reviewed by an editorial staff to maintain consistency of language across

the items and content areas. Items were also reviewed by Florida teachers at a content review session

and the graphics for the items were reviewed by a vision specialist at the Florida Department of

Education.


3.5 Florida Alternate Assessment Field Test: Bias and Sensitivity Review

Issues of bias in test materials are of particular concern because an important tenet of

assessment is to ensure that all students have an equal opportunity to demonstrate their knowledge

and skills. The Bias and Sensitivity Review Committee members met once in May 2007 prior to

field test. At this meeting, the committee had two tasks. The first task was to review the Bias and

Sensitivity Guidelines for the Development of the Florida Alternate Assessment. The second task was

to review the reading passages and a few initial science items to determine if they were likely to

place a particular group of students at an advantage or disadvantage for non-educational reasons.

Any passages that were identified as biased were either revised or removed.

The Bias and Sensitivity Review Committee consisted of nine individuals selected to

participate by the Florida Department of Education (see list in Appendix A). They included three

special education teachers, one special education administrator, two vision specialists, one bilingual

assessor, one assistive technology specialist and one school psychologist. The meeting was held via

video conference. The Measured Progress special education specialist and lead developers for

reading and science were also present along with the alternate assessment department staff from

Florida. The meeting began with a review of the Bias and Sensitivity Guidelines. Emphasis was

placed on the accessibility of the reading passages and science items for the population of students in

alternate assessment.

Recommendations for the science items centered on using common classroom items in the

graphics provided for the items whenever possible. This concern was raised as a way to help address

issues of assessment administration for students with visual impairments who might need to use

actual objects during testing. Participants also made recommendations in some of the science topic

areas as to activities or objects students would have more familiarity with. For example, more

students would have opportunities to use microwaves then ovens.


For the reading passages, participants reviewed the passages and associated passage graphics.

Recommendations were made to keep graphics uncluttered by removing horizons and background

that did not need to be part of the graphics. Heavier line weights were recommended and if the detail

in a graphic distracted, recommendations were made to simplify the graphics.

For the passage topics panelists made recommendations on topics that students would have

the opportunity to be involved in. For example, a passage about snow might not be the best for

Florida students. Panelists also made recommendations when they felt a particular portion of the

passage showed bias toward a certain disability group, such as students with low hearing or low

vision. One final area of recommendations was sensitivity to experiences students may have had,

staying away from passages where students were laughed at or bullied. Overall the majority of

passages were accepted as is, a few were slightly revised and a few were thrown out altogether. All

information from the bias meeting was compiled and passages were marked as accepted or rejected

and any revisions were noted. This record was shared with the DOE staff.

3.6 Florida Alternate Assessment Field Test: Content Review

The content review for all items took place July 16-18, 2007 in Tampa. Panels were

convened for each of the four content areas, reading, writing, mathematics and science. Each

committee consisted of elementary, middle school and high school special educators and content

area educators. (See Appendix A for the list of panelists.)

Panels were facilitated by the lead developer for each of the content areas and two special

education specialists were also present to assist in each of the rooms as needed. Panelists were asked

to review the items to make sure that they addressed each of the Access Points, to review the

Teacher Will column for clarity on what the teacher needed to do and say, to make sure that there

was only one correct answer, to review the graphics within each of the items for clarity and to be


aware of the difficulty levels within items (from participatory to independent) as well as across the

grade levels. Recommendations by the panelists were written on each of the items.

After the panelists completed their content area review the developers, special education

specialists and department staff met to review the panelists‘ recommendations and make final

decisions on each of the items. The recommendations centered on simplifying the graphics,

rearranging some of the items where the supported level of complexity seemed more difficult than

the independent level of complexity, and reducing the complexity of the materials and/or distracters.

Overall, the content review went well and teachers were pleased to be a part of the process.

Feedback for each of the content review panels is compiled in Appendix C.

3.7 Florida Alternate Assessment Field Test: Administration

The Florida Alternate Assessment field test consisted of items for Reading and Mathematics

in grades 3-10; Writing in grades 4, 8, and 10; and Science in grades 5, 8, and 11.

Table 3-12. 2007-08 Florida Alternate Assessment: 2007 Field Testing by Grade Level and Content Area

Grade Reading Mathematics Writing Science 3 X X 4 X X X 5 X X X 6 X X 7 X X 8 X X X X 9 X X 10 X X X 11 X

The field test was administered October 29 through November 9, 2007 to approximately 500

students in each grade level In order to maximize school district and teacher participation in the

field test each district‘s proportion by grade of alternate assessment participants in Spring, 2007 was

calculated. This number was then stratified within district and grade on the following variables:

gender, race/ethnicity, and level assessed in Spring, 2007. Students were tested in all content areas


assessed for that grade level. The field test was composed of 30 items in each content area. Each

item set was composed of three Access Points representing increased levels of complexity:

participatory, supported, and independent. The student who answered correctly at the participatory

level moved up to the supported level of the item set. If the supported level was answered correctly,

the student was administered the independent level question. For example, fourth-grade students

were assessed in reading, mathematics, and writing. Each content area test was composed of 30 item

sets. With the possibility of being administered all three levels, a student might receive up to 90

questions per area and up to 270 questions across all contents.

Table 3-13. 2007-08 Florida Alternate Assessment: Field Test Participants by Grade Level

Num

ber

of S

tude

nts

600 500 400 300 200 100

0

459 480 480 468 463 460 453 472 370

03 04 05 06 07 08 09 10 11

Grade-Level

Assessment administration training was provided to teachers involved in the field test prior to

the field test. Measured Progress staff presented two one-day training sessions to approximately 120

Alternate Assessment Coordinators. They were provided with a train-the-trainer administration

manual. This train-the-trainer session was provided prior to the field test so that trainers could train

all teachers involved in the field test and then provide further training for all teachers later in the

year. Details of the training provided are described in section 4.1.

3.8 Florida Alternate Assessment Field Test: Statistics

The field test results were analyzed to provide information for the selection of items to be

utilized on the 2007-08 operational assessment. The analysis included the p-values (difficulty level)


for each level of complexity of an item, discriminations (item correlations with total score), and DIF

values (for males versus females). DIF results showed no significance for any of the items.

Therefore, selection was based on p-values and discriminations and survey feedback (see next

section) on: item specifics, graphics within items, the amount of teacher materials to be gathered, the

number of cutouts required by items, and the balance of items required by the blueprints at each

grade and content area.

3.9 Florida Alternate Assessment Field Test: Survey Results

An on-line survey was provided during the field test administration window and for one

week after the close of the window. Approximately 2000 teachers responded to the on-line survey.

The survey consisted of three parts. The first part requested background information that included

teacher demographic information, whether or not the teacher was instructing the students using the

Access Points, and whether or not the teacher had attended the administration training. This first part

also asked for feedback on the administration process including the clarity of the administration

manual directions, the ease of the administration process, the ease of use of the materials required for

the assessment, and the appropriateness of the graphics included in the items. The second student

specific part asked for student demographic information, and specifics as to how that student

interacted with the assessment, such as whether the content was new to the student and whether the

format of the administration was new to the student. The third part of the on-line survey was a

voluntary portion and allowed teachers to respond to specific items and any issues that they had with

the items. There were not more than 5 to 10 responses for any one item.

Around 50% of the respondents stated that they were consistently instructing their students

utilizing the Access Points in both English language arts and mathematics. Ninety-three percent of

the teachers that responded had attended administration training and 81% felt that they had received

enough training to administer the field test. Around 70% of respondents felt it was clear how to


administer the items. Forty-seven percent of respondents felt the graphics were appropriate for the

items, while 43% did not feel the graphics were appropriate. Over 80% of the respondents felt that

the auxiliary materials that needed to be cut out were difficult to organize. Other themes from the

survey indicated that the amount of time for administration was too long, that the cutting out and

organizing of the materials took a significant amount of time, and that students needed to hear the

item prompt more than one time. Survey results can be found in Appendix C.

3.10 Florida Alternate Assessment: Revisions Based on Field Test Results

Field test survey results were reviewed by both Measured Progress and Department of

Education staff. Results were also shared at a meeting of the Alternate Assessment Advisory on

November 29, 2007. The DOE requested their feedback and recommendations on the length of the

administration of the assessment based on the number of items, the types of graphics to use within

the assessment, and the format of the assessment and auxiliary materials. The advisory

recommended that the operational assessment consist of between 16 and 20 items only. This issue

was also shared with the Florida Technical Advisory Committee (TAC) earlier in November for their

input. Because item statistics were not available at the time, the TAC did not feel confident in

recommending a particular number of items for the assessment. They did note that the number of

items should match with the intent and type of information that Florida wanted to report out for their

alternate assessment. The assessment was reduced from a 24 item design to a 16 item design. The 24

item design was initially proposed to allow for the reporting out of information at the standard level,

however, based on conversations with the FLDOE, the Alternate Assessment Advisory and the

Florida Technical Assistance Advisory it was made clear that reporting out at the content area level

was the overall intent of the Florida Alternate Assessment.

Regarding item graphics, some of the advisory members felt they should be more —real-life“;

whiles some of the advisory members felt that the Picture Exchange Communication System


(PECS), with their heavy lines and symbolic nature, were better. Their recommendation was not

conclusive.

The format that required teachers to cut out all of the auxiliary materials was recognized as a

major issue early in the field test window, so Measured Progress and Department staff had already

collaborated on a solution that was presented to the advisory committee. The materials would be

provided in Auxiliary Materials Booklets on 11 inch x 17 inch paper, bound on the 11 inch side.

Each level of the item would be presented on a separate page and would be positioned on the page

either horizontally or vertically. If the item included a stimulus card, it would be separated from the

response cards with a heavy black line. Each page of the booklet would be numbered with the grade,

level and item number in the upper right-hand corner (e.g., 3P-1 or 3S-1 or 3I-1, where 3 represented

grade3; P, S, and I represented Participatory, Supported and Independent levels, respectively; and 1

represented item number 1). This would allow for ease of use by the teacher during administration.

A minimal amount of items in Reading, Mathematics, and Science would still require cutouts

due to the nature of the item; for example, an item might require that the student arrange three

pictures in sequence. The only content area that would be composed entirely of cut outs would be the

writing test, and cards and strips would be pre-cut by the printer and no longer require teacher time

for cutting. A sample item of this new format can be found in Appendix D.

The advisory was presented with a sample of the above format and felt that it would address

the concerns that teachers had voiced based on the field test administration. It was also decided that

the administration manual would be updated to clarify test administration procedures, give

information on how to orient students to the assessment, better define and provide directions around

prompting and cueing, and clarify allowable adaptations for the alternate assessment.


32

Chapter 4. TRAINING AND ADMINISTRATION

4.1 Administrator Training

4.1.1 Professional Development

A train-the-trainer model workshop was provided by Measured Progress to approximately

120 individuals (two trainings of 60 individuals each) in early September 2007. Another 212

individuals were trained through the Accountability and Assessment for Students with Disabilities

(AASD) project in late September 2007. These trainers then trained approximately 3709 additional

teachers prior to the administration of the field test.

The train-the-trainer workshop was provided by the Measured Progress special education

specialist who had been involved with the development and review of the alternate assessment items

and had written the administration manual. The training included an overview of the administration

manual, a review of all of the activities prepared for training teachers, a review of the sample items

included in the manual and a review of the items prepared for the trainers. The PowerPoint that was

utilized for the training included a detailed notes section that directed trainers on what to say and

how to present the training. (See Appendix C for feedback related to the train-the-trainer sessions.)

Following the train-the-trainer sessions an updated administration manual was sent out for all

trainers and teachers involved in administration of the alternate assessment. The Department also

provided each teacher a DVD showing the administration of items using the new format of the

assessment materials.

4.1.2 Administration Manual

The administration manual includes sections that outline the assessment and its purpose, the

participation criteria for the assessment, the general administration procedures and materials of the

assessment, the content specific directions needed for the assessment, the scoring rubric and

directions on how to score each item in the assessment, how to fill out the student answer document,

Chapter 4—Training and Administration 33 2007–08 Florida Alt Technical Report

sample items and criteria and allowable accommodations for specific sectors of the student

population.

The Field Test Administration Manual was revised and an Operational Administration

Manual was provided to trainers and teachers in January 2008. This revised manual addressed the

changes to the assessment materials format. It also clarified some administration issues that were

raised through the field test. It was clarified that teachers could repeat the item two times, for a total

of three times, to a student as needed. It was also clarified that in items where the student was asked

to respond with more than one answer that the student could be prompted to finish an answer. For

example, an item might ask a student to sequence three phrases, the manual revisions clarified that it

was allowable for the teacher to prompt —What comes next?“ until the student had finished

sequencing all three phrase. The manual also added some appendices, including a list of the items

that required teacher-gathered materials and what those materials are. This allowed teachers to begin

gathering materials prior to the assessment administration window.

4.1.3 Training DVD

The Florida Department of Education developed a half-hour long DVD using the new

administration manual and newly formatted sample items. These DVDs were provided to school

districts on disc and made available for video streaming on the FLDOE website in January 2008.

The DVDs include a variety of students being administered items. The item administrations include

students where they move through an item all the way to the independent level and those that require

scaffolding at the participatory level of the item. They also highlight the changes in administration,

such as the repeating of the item prompt and the focusing of the student on the assessment materials.


4.2 Operational Test Administration

The Florida Alternate Assessment consisted of 16 items each in Reading and Mathematics in

grades 3-10; Writing in grades 4, 8, and 10; and Science in grades 5, 8, and 11. The operational test

was administered between February 15 and April 2, 2008 to approximately 2200 to 3000 students in

each grade level. See the Figure 4-1 for the number of students by grade level assessed.

Florida Alternate Assessment Spring 2008 Operational Test

Number of Students Assessed by Grade-Level

2388 2478 2519

2971 3078 3033

2280 2310

2234

0 500

1000 1500 2000 2500 3000 3500

03 04 05 06 07 08 09 10 11 Grade-Level

Num

ber o

f Stu

dent

s

Figure 4-1. 2007-08 Florida Alternate Assessment: Number of Students Assessed by Grade Level

Students were tested in all content areas assessed for each grade level. The assessment was

composed of 16 items in each content area. For each item there are three Access Points representing

increased levels of complexity, participatory, supported, and independent. The student who answers

the participatory level correctly moves to the supported level for that item. If the supported level was

answered correctly, the student was administered the independent level question. For example, a

fourth-grade student was assessed in reading, mathematics, and writing. Each content area test was

composed of 16 items with the possibility of administering all three levels, for a total of 48 questions


per content area. Depending on student performance, a total of 48 to 144 questions were

administered in grade 4.

4.2.1 Operational Test Survey Results

An on-line survey was provided during the administration window and for one week after the

close of the window. Approximately 850 teachers responded to the on-line survey. It is unclear the

number of total teachers that are involved in administering the assessment, however, 8000

administration manuals were distributed. The survey consisted of three parts. The first part requested

background information that included teacher demographic information, whether or not the teacher

was instructing the students using the Access Points, and whether or not the teacher had attended the

administration training. This first part also asked for feedback on the administration process

including the clarity of the updated administration manual directions, the ease of the administration

process, the ease of use of the materials required for the assessment based on the new format, and the

appropriateness of the graphics included in the items. The second part asked for student

demographic information, and specifics as to how that student interacted with the assessment, such

as how long the administration of each content area took for the student and the number of breaks

the student required during administration. The third part of the on-line survey was a voluntary

portion and allowed teachers to respond to specific items and any issues that they had with the items.

Once again there were not more than 5 to 10 responses on any one item.


Around 50% of the respondents stated that they were consistently instructing their students

utilizing the Access Points in both English language arts and mathematics. Three hundred and fifty-

four of the five hundred teachers who had also participated in the field test responded that they had

attended an additional administration training and 96% of those who attended additional training felt

that they had received enough training to administer the new format of the assessment. Of those 500

teachers who had also participated in the field test 70% felt it was clear how to administer the items,

and 73% felt the graphics were appropriate for the items. About 93% of those same respondents felt

that the materials were more manageable and 90% stated that it was easier to organize the materials.

When comparing the amount of time that the reading test took to administer, with the change in the

number of items from 30 to 16, the amount of time it took was reduced. On the field test only 54%

of respondents were able to administer the test in 3 hours or less, while on the operational test 84%

were able to administer the test in 3 hours or less. Survey results can be found in Appendix C.


Chapter 5. SCORING

5.1 Decision Rules for Scoring

In order to receive a valid score for a grade-relevant content area, all 16 items must be

completed correctly; this is regardless of whether a reason for not being assessed is bubbled on page

1 of the Student Answer Sheet (i.e., Took FCAT, Withdrew, Home school, etc.) or the ”Student

Took FCAT [content area]‘ bubble for a particular content area (page 2 of Student Answer Sheet) is

filled in.

The following list describes situations where a valid score for a specific content area cannot

be achieved:

° Invalid Bubble Completed- A total score cannot be calculated for any content areas

(complete or incomplete) on an Answer Sheet marked as —Invalid“. The Invalid bubble is

located on the bottom of page 1 of the Student Answer Sheet. Teachers are asked to mark

the Invalid bubble if the answer sheet is defective, soiled, or incorrectly completed.

° Missing Student Grade - A total score cannot be calculated for any content areas

(complete or incomplete) on an Answer Sheet where student grade has not been marked.

° Incomplete Content Area - A total score cannot be calculated for a content area unless

all 16 items have been completed. Content areas with less than 16 bubbled items are

labeled as NS (i.e., No Score œ not enough data to calculate a score).

° Multiple Responses Bubbled for an Item - A total score cannot be calculated for a

content area if more than one answer has been bubbled in for any one item. An item-level

score cannot be determined if an item has more than one answer. The content area is

therefore labeled as NS (i.e., No Score œ not enough data to calculate a score).

° Content Area not Assessed - For partially completed Student Answer Sheets (i.e., grade

3 student - only Reading content area completed), grade-relevant content area(s) that

were not completed are labeled as NA (i.e., Not Assessed).

See Figure 5-1 for a visual depiction of the scoring decision rules process.

Chapter 5—Scoring 39 2007–08 Florida Alt Technical Report

Form Level Decision Was the INVALID bubble filled in?

Content Area Level Decision

Was the student‘s grade level bubbled in?

Were any of the 16 items for the content area completed?

Were the 16 items completed correctly (i.e., only 1 response

bubbled in per item)?

No Yes

Record removed from scoring

Record removed from scoring

Were all 16 items for a given content area bubbled?

NSNA TOTAL SCORE NS

No Yes

No Yes

No YesNo Yes

Figure 5-1. 2007-08 Florida Alternate Assessment: Scoring Decision Rules Flowchart for Grade-Relevant Content Areas

Most records that are removed from scoring count as participating but do not receive an

assessment score. Following is a table indicating the number of valid scores, no scores and not

assessed for the 2008 operational assessment by content area. Less than 1% of the total content area

tests were deemed as no scores and less that 1% of them were deemed as not assessed.


Table 5-1. 2007-08 Florida Alternate Assessment: Overview of Assessment Outcomes by Content Area

Assessment Outcomes By Content Area Math Writing Science Reading

Valid Score 20,012 19,973 7,620 7,787

NS (No Score) Multiple Responses Bubbled for an Item 41 31 12 19

NS (No Score) Incomplete Content Area 133 111 98 36

NA (Not Assessed) 72 143 147 98

5.2 Scoring Rubric

Each item is scored by the teacher during the administration process. The Student Test

Booklet provides a place to mark the score that the student has earned on each item during

administration. The teacher then needs to transfer the final score for each item to the student answer

document. Teachers may also record the student scores on each item directly on the student answer

document during administration if they prefer. Students may only earn a single score point for each

item. Please see section 3.1 for a detailed description of this process. Following is the scoring rubric

that was utilized during the administration process.


Scoring Rubric


5.3 Scoring Process

5.3.1 Handling of Incoming Forms

Incoming Shipments

° Incoming shipment information is entered into a Florida Alternate Assessment

management database as shipments arrive. Barcodes from light blue TO BE SCORED

labels affixed to incoming boxes and courier tracking numbers are scanned into the

database, along with the name of the sending district, and the date of arrival. Each box

includes separate TO BE SCORED materials envelopes from each school returning

answer sheets for scoring. School envelopes include student answer sheets and a

Document Count and Return Summary Form. A blue label with a unique barcode

identifying the returning school is affixed to the front of each envelope. When boxes (or

packages) are opened, the barcode on each envelope‘s label is scanned into the

management database. Each envelope barcode is linked to the barcode on the box in

which it arrived.

° Depending on size, packages are either locked in a cabinet or stored in a separate locked

office before processing.

° Since processing of packages is done on a by-district basis, only boxes/packages for the

relevant district are moved to the processing area at a time.

Document Sorting

° TO BE SCORED materials are separated into 4 separate trays by district: 1) Completed

Student Answer Sheets; 2) Blank/Unused Student Answer Sheets with no demographic or

item-level data; 3) Document Count & Return Summary Forms; 4) Other miscellaneous

materials (e.g., business cards, post-it notes, student records) œ these other materials are

reviewed by supervisors and either stored or destroyed.

° All documents are removed from packaging. As a safety measure, all empty envelopes

are re-inspected once forms have been removed to ensure that no forms remain in the

envelopes.

° If additional notes from district coordinators or examiners are discovered (e.g., —DO NOT

SCAN“), the notes and corresponding answer sheets are shared with supervisors before

proceeding.


° Additional staples and paperclips are removed from forms.

° Completed forms are checked for missing district numbers and/or school numbers as they

are processed:

o If either of these items is missing, the information is added only if the correct district/school number can be discerned from the envelope label or Document Count & Return Summary Form. Staff are trained to ask supervisors for assistance, whenever necessary.

° Student Answer Sheets and Document Count & Return Summary Forms are stored in

locked cabinets (separated by district) for the next stage of processing.

° After opening all boxes/packages for a particular district, staff members date and initial

next to the district‘s name in a processing log.


Chapter 6. SCANNING

Scan Station is the Teleform module used to capture data and form images from the Student

Answer Sheets. Once forms have been scanned, the Teleform system evaluates the data captured,

which are subsequently verified by a Verifier Station operator.

Scan Station operators perform the following steps:

1. Log in 2. Remove any remaining staples and paperclips from the forms 3. Create batches no thicker than 1“ (approx. 40 forms) 4. Flip through forms to help break up stack 5. Place forms in scanner bay 6. Select New Batch under the File menu of Batch Explorer 7. Select Job-FLALT 8. Confirm under the Processing Tab that Setting reads, —Panasonic“ and —Feeder-

Front & Back“ 9. Click —Start“ 10. Watch for errors as images are scanned

Quality Check

° If multiple pages are scanned together, lines appear, or if other imaging issues occur,

operators are instructed to follow the steps below:

11. Stop scanning by removing forms from scanner bay 12. Find forms in which errors occurred 13. Delete the corresponding images in which errors occurred 14. Rescan forms that were deleted

° If batch is not yet complete, load remaining forms in scanner and select —Continue“.

° When a batch is complete, review images in Batch Explorer - if an error is detected,

follow steps 1-4 above.

° If quality of images is acceptable, —Accept“ batch.

° Batch will appear in Batch Explorer as —Ready to Evaluate“.

Post Processing

° Batch number is written on a batch cover sheet

° Batch cover sheet is placed on top of scanned batch

° Batch and cover sheet are bundled with a rubber band

° Date, district number, and initials are noted in the batch log for each batch number

Chapter 6—Scanning 45 2007–08 Florida Alt Technical Report

° Batches are placed in a locked cabinet for Verifier operator to review

° Once all of the forms for a district have been scanned, operators date and initial next to

the appropriate district name on the scan log provided

° Operators log out of scan station when they switch stations or once scanning has been

completed for the day

Cleaning

° The scanner is cleaned after every 25 batches or whenever images show stray

streaks/lines œ staff members date and initial next to the appropriate batch in the batch log

once they have cleaned the scanner.

° Scanner is opened from the front and rollers are cleaned of debris using isopropyl alcohol

and cotton swabs or wipes.

° Compressed air removes dust, residue, and staples.

Verifying & Committing Data

° Teleform Verifier operators perform the following steps:

15. Log in using secure User ID & Password 16. From the —Utilities“ menu, operators select —Batch Management“ 17. Click on a batch to begin 18. Retrieve the matching, hard copy batch of original Student Answer Sheets from

the locked cabinet 19. Once a batch is selected, the digital image of each Student Answer Sheet will

appear for verification, if operator review is required.

Verifying Demographic Information

° To ensure the accuracy of demographic information provided on the Student Answer

Sheets, the following elements were programmed into the system:

° 1) The Verifier module automatically forces the operator to stop and review all

demographic fields on non-pre-identified (i.e., handwritten) Student Answer Sheets.

° 2) Demographic information on page 1 of the pre-identified Student Answer Sheets are

not verified. Each pre-identified Student Answer Sheet is tied to the corresponding

Survey 2 database record using the unique ID (P-LINK) on the bottom, left-hand corner

of the form. Upon export, a SQL database trigger updates the record with the pre-

identified demographic data.


° 3) The system is programmed to automatically stop at all fields completed in the —Student

Demographic Information Corrections“ section on page 2 of ALL Student Answer Sheets

(i.e., pre-identified or non-pre-identified).

° When the Verifier module stops on a demographic data field, the operator must determine

if the system‘s Intelligent Character Recognition (ICR) deduction is correct or if there is

an error that needs to be corrected.

o If the system has read the intended character correctly, the operator accepts the system‘s inference by moving on to the next field.

o If the system interprets a character erroneously, the operator corrects the error by typing in the correct character based on the actual information written on the scanned image or hard copy of the form.

o Similarly, if the system interprets a stray mark as a character, the operator deletes the unnecessary characters.

° If a field value does not meet certain predetermined criteria, operators can either confirm

and accept the —Out of Range“ values or they can skip to the next field, which leaves the

field flagged for review by supervisors later on.

° Operators are trained to enter characters exactly as they are found on the forms. Their

principal mission is to recreate the data from the original form precisely as the data was

intended.

Verifying Item-Level Data

° Multiple & Inconclusive Responses

The system is programmed to identify assessment items where: a) more than one answer

has been completed; b) Teleform Verifier was inconclusive about whether an answer has

been bubbled. As the operator toggles through the Student Answer Sheets, a Field

Violation message box will appear (when the system locates an instance of case a. or b.

above) asking the operator, —Can you identify the correct bubble?“

o If the operator can clearly discern which value the examiner intended to submit, then they correct or confirm the value and submit.

o If the operator CANNOT tell which value the examiner intended to submit, then they write the P-LINK, content area, and error type on the batch cover sheet for supervisors to review. The original forms are then pulled and placed at the top of the batch.

° Missing responses

The system is also programmed to count the number of items with responses for each


grade-relevant content area (e.g., only Science for 11th grade). If the total number of

counted responses does not match the total number of items for a content area (i.e., 16

items), then a flag is raised and the system will automatically stop on the incomplete

item(s). Verifier operators are trained to review the original Student Answer Sheet (rather

than the scanned image) to determine whether an item has, in fact, been completed. If any

item is blank for a grade-specific content area, the operator writes the P-LINK, content

area, and error type on the batch cover sheet for supervisors to review. The original forms

are then pulled and placed at the top of the batch.

Missing Pages

° If Verifier identifies a form as having a missing page, the operator deletes the form image

from the system and pulls the hard copy from the batch for rescanning. Verifier also

identifies forms that may have unidentified pages due to page overlap during scanning,

stray marks, torn forms, or damage to square cornerstone markers. These forms are also

rescanned.

Committing Batches to the SQL Server Database

° Once the batches have been verified, they are transferred to a supervisor for quality

checking.

° The front cover of each batch is checked by the supervisor for errors noted by Verifier

operators.

o If the batch cover sheet contains errors found (e.g., more than one answer has been bubbled for an item), the supervisor reviews the original Student Answer Sheets to confirm these errors.

° When the supervisor confirms that an error was, in fact, submitted by the examiner,

he/she initials the cover sheet next to the location where the error was noted.

° If an error is determined to be a false positive, the supervisor will correct the item in

Teleform Verifier, make a note of the change on the batch cover sheet, and sign and date

the cover sheet where the change is noted.

° All Student Answer Sheets where the system has identified errors have a status of —Needs

Review.“ A batch cannot be committed until the status of all Student Answer Sheets is

—Evaluated OK.“


° Supervisors randomly check 5 Student Answer Sheets per batch where errors were not

flagged by the system.

° The batches can then be committed to the database. The supervisor signs off that the

batch has been committed.

6.1 Data Security

Individuals are only granted permission for actions needed to perform their jobs. Limiting

actions to those properly authorized protects the confidentiality and integrity of data within the

processing environment. All employees are required to sign a confidentiality agreement.

6.2 Electronic Records

All authorized personnel have individual usernames and passwords to access the stand-alone

network, which stores secure student data. If personnel leave their computers for more than two

minutes, a password-protected screen saver is activated. A very limited number of employees have

access to sensitive electronic records. All sensitive electronic records, including scanned answer

sheet images, assessment data, and student demographic information are stored on the SQL server

and backed up every night.

All electronic records are protected from unauthorized access while in storage and while

being processed through the use suitable information security techniques, such as password-

protection and analogous methods. Access control mechanisms must also be utilized to ensure that

only authorized users can access data to which they have been granted explicit access rights.

Additionally, any computer and/or electronic devices where these electronic records reside, such as

database servers, local hard drives, external hard drives, tape or optical backups, etc., are always

kept within secure premises, as described below.

Authorized individuals are trained to avoid transmitting sensitive data through electronic

means proven to be easily intercepted and/or modifiable, such as unencrypted email communications


or unsecured FTP connections. Transmission of sensitive information via facsimile documents is

also prohibited.

6.3 Physical Records

Only authorized employees have access to student data for processing purposes. Each

employee must ensure that confidential data under their direction or control are properly labeled and

safeguarded according to their sensitivity and criticality. All physical records must be kept in full

view by the authorized employees while being accessed and/or processed, and properly stored and

secured if the premises are left for any period of time. Sensitive physical records are stored in locked

cabinets, and only supervisors have access to their keys.

Location Specifications

The premises where sensitive physical and electronic records are stored are protected at all

times from unauthorized access, through a combination of building security access systems, security

personnel, and suitable locks in doors and any other similar points of access. Storage and filing

cabinets are also protected by locking mechanisms, independently from any additional access control

to the rooms where they are located. Building windows are fixed panes made of impact-resistant

glass that do not open. The building‘s security access system limits access to the building after hours

and during weekends. An Access Card is required to gain entry to the building when the security

system is activated. The premises are also protected by a security company, which provides a

security guard 24hours a day, 7 days a week.

6.4 Data Disposal

Both physical and electronic records are destroyed, deleted and/or purged through any

number of means that guarantee the technical impossibility of these records being recovered, be it

partially or completely. Any backup copies of electronic records that might exist, regardless of

format, are also disposed of accordingly. Data assets, both physical and electronic, are kept for the


period of time considered as mandatory by any applicable laws. If any data assets were to fall

outside the scope of these periods of time, then all necessary steps for their disposal are taken.

6.5 Secure Test Material Distribution & Return

All test material shipments to and from the districts are shipped using tracking mechanisms.

Every district and school materials box within a district shipment contains a label with an internal

scannable barcode as well as a standard courier/freight shipping label. For tracking purposes,

internal and shipping barcodes are stored in a management database before shipments are picked up

by couriers. Every district shipment includes school-level and district-level packing lists detailing all

of the materials included. For districts receiving pallets of materials, a pallet map is also provided,

describing how many cartons are included for each school and the skid numbers where the cartons

can be found.

Both district and school test coordinators are instructed to inventory shipment contents within

24 hours of receipt and report any discrepancies immediately. Once secure test materials arrive at the

districts, district assessment coordinators are responsible for storing these materials in secure, locked

facilities. It is the responsibility of district assessment coordinators to ensure that materials are

handled appropriately during distribution to and return from schools. Likewise, school test

coordinators are instructed to store test materials in secure locations. Materials are shipped using

DHL, UPS, or AAA Cooper only; the type of courier is determined based on type and quantity of

materials. All shipments to districts are tracked to ensure delivery by a specific date.


Chapter 7. REPORTING

7.1 Report Shells

Reports are generated at the following levels:

State-level report œ contains the number of students assessed and percentages of students

scoring at each level of complexity (i.e., Participatory, Supported, Independent) and level of

performance (i.e., Basic, Proficient, Advanced) for each district as well as the State‘s overall results

by content area.

District-level reports œ contain the number of students assessed and percentages of students

scoring at each level of complexity (i.e., Participatory, Supported, Independent) and level of

performance (i.e., Basic, Proficient, Advanced) for each school in a given district as well as the

district‘s overall results by content area.

School-level reports œ includes the list of students assessed in a given school along with their

level of complexity (i.e., Participatory, Supported, Independent), level of performance (i.e., Basic,

Proficient, Advanced), and total score by content area. The report also contains a summary of the

school‘s overall results.

Student-level reports œ includes the student‘s basic demographic information (e.g., name,

grade, school, etc.), total score, level of complexity (i.e., Participatory, Supported, Independent), and

level of performance (i.e., Basic, Proficient, Advanced) by content area. See Appendix E for sample

report shells.

In addition to the reports listed above, parent and teacher brochures were prepared to be

returned with the individual student reports. The parent brochures focus on providing an overview of

the Florida Alternate Assessment including the Access Points and a description of the levels of

complexity, information on who determines whether or not the student will participate in alternate

assessment, when the assessment takes place, who administers the assessment and how the results

are used. The teacher brochure includes some of the same information, but focuses more on what

Chapter 7—Reporting 53 2007–08 Florida Alt Technical Report

results are provided and how they can be utilized by the teacher. (Copies of the brochures can be

found in Appendix F.)

7.2 Decision Rules for Reporting

° Reports are not generated for students where no items in the content area(s) specific to

the student‘s grade are completed.

° Data scanned from Student Answer Sheets marked as —Invalid“ are not included in

reports. The Invalid bubble is located on the bottom of page 1 of the Student Answer

Sheet. Teachers were asked to mark the Invalid bubble if the answer sheet was defective,

soiled, or incorrectly completed.

° Data scanned from Student Answer Sheets where no grade level is indicated are not

included in reports.

° Reports are not generated for students where deceased is indicated as the Reason Not

Assessed (page 1).

Chapter 7—Reporting 54 2007–08 Florida Alt Technical Report

SECTION III—TECHNICAL CHARACTERISTICS OF THE FLORIDA ALTERNATE ASSESSMENT

Chapter 8. TECHNICAL CHARACTERISTICS OF THE FLORIDA ALTERNATE ASSESSMENT

Highlights:

Score Distributions Across Grade Levels

¢ Score Means ¢ Score Standard Deviation (scale range 0 œ 144)

£ Mathematics £ Mathematics ¢ 69.0 œ 81.1 ¢ 34.8 œ 38.6

£ Reading £ Reading ¢ 71.9 œ 91.7 ¢ 36.3 œ 41.2

£ Science £ Science ¢ 88.8 œ 96.5 ¢ 40.4 œ 43.5

£ Writing £ Writing ¢ 76.6 œ 81.5 ¢ 36.5 œ 40.6

8.1 Performance Standards and Student Results

Standard setting activities for the Florida Alternate Assessment in reading, writing,

mathematics, and science occurred in two stages: the preliminary stage occurred the week of April

7œ11, 2008 and the verification stage took place the week of July 15œ18, 2008. At the April standard

setting meeting, preliminary cut-points were recommended for each grade and content area using

data from the Fall 2007 Florida Alternate Assessment field test administration. The cut points from

the preliminary stage were used for the Spring 2008 reports. Complete documentation of the

standard setting procedures can be found in <insert links for both manuals here>. Table 8-1 presents

the final raw score ranges for performance levels by grade-content and level of complexity.

Chapter 8—Technical Characteristics 55 2007–08 Florida Alt Technical Report

Grade

Table 8-1. 2007–08 Florida Alternate Assessment: Score Ranges for Performance Levels by Grade, Content, and Level of Complexity

Performance Level Content Area

Level of Complexity Basic Proficient

Participatory 0œ27 28œ47 Advanced

48œ62

3 Independent Participatory

99œ108 0œ19

109œ121 20œ29

122œ144 30œ57

Independent 99œ114 115œ129 130œ144

Supported 63œ72 73œ87 88œ98 Reading

Supported 58œ70 71œ86 87œ98 Mathematics

Participatory 0œ27 28œ47 48œ62 Supported 63œ73 74œ87 88œ98 Reading Independent 99œ109 110œ122 123œ144 Participatory 0œ20 21œ32 33œ57 Supported 58œ70 71œ86 87œ98 Mathematics 4 Independent 99œ113 114œ129 130œ144 Participatory 0œ22 23œ35 36œ63 Supported 64œ69 70œ84 85œ98 Writing Independent 99œ112 113œ128 129œ144 Participatory 0œ27 28œ46 47œ62 Supported 63œ73 74œ87 88œ98 Reading Independent 99œ110 111œ123 124œ144 Participatory 0œ22 23œ34 35œ57 Supported 58œ70 71œ86 87œ98 Mathematics 5 Independent 99œ113 114œ129 130œ144 Participatory 0œ26 27œ45 46œ61 Supported 62œ73 74œ87 88œ98 Science Independent 99œ114 115œ127 128œ144 Participatory 0œ27 28œ46 47œ62 Supported 63œ73 74œ87 88œ98 Reading


99œ111 0œ24

112œ124 25œ37

125œ144 38œ57

Independent 99œ112 113œ128 129œ144 Supported 58œ69 70œ86 87œ98 Mathematics

Participatory 0œ27 28œ45 46œ62 Supported 63œ73 74œ86 87œ98 Reading


99œ112 0œ25

113œ126 26œ39

127œ144 40œ57

Independent 99œ112 113œ128 129œ144 (cont‘d)

Supported 58œ69 70œ86 87œ98 Mathematics


Content Level of Performance Level Grade Area Complexity Basic Proficient Advanced

Participatory 0œ28 29œ45 46œ62 Supported 63œ73 74œ86 87œ98 Reading Independent 99œ113 114œ127 128œ144 Participatory 0œ27 28œ42 43œ57 Supported 58œ69 70œ86 87œ98 Mathematics


99œ111 0œ24

112œ128 25œ40

129œ144 41œ63

Independent 99œ111 112œ126 127œ144 Supported 64œ71 72œ86 87œ98 Writing

Participatory 0œ26 27œ44 45œ61 Supported 62œ72 73œ87 88œ98 Science Independent 99œ115 116œ128 129œ144

Participatory 0œ28 29œ44 45œ62 Supported 63œ73 74œ86 87œ98 Reading


99œ114 0œ29

115œ128 30œ44

129œ144 45œ57

Independent 99œ111 112œ127 128œ144 Supported 58œ69 70œ86 87œ98 Mathematics

Participatory 0œ28 29œ44 45œ62 Supported 63œ73 74œ86 87œ98 Reading Independent 99œ115 116œ129 130œ144 Participatory 0œ30 31œ47 48œ57 Supported 58œ68 69œ87 88œ98 Mathematics 10 Independent 99œ110 111œ127 128œ144 Participatory 0œ25 26œ43 44œ63 Supported 64œ73 74œ87 88œ98 Writing Independent 99œ111 112œ125 126œ144

Supported 62œ71 72œ88 89œ98 Participatory 0œ26 27œ43 44œ61

11 Science Independent 99œ115 116œ129 130œ144

Table 8-2 presents the performance level frequency distributions on the 2007-08 Florida

Alternate Assessment by grade and content. (Cumulative raw score frequency distributions for each

2007-08 Florida Alternate grade and content area may be found in Appendix H.)


Table 8-2. 2007–08 Florida Alternate Assessment: Performance Level Counts and Percentages by Grade, Content, and Level of Complexity

Content Performance Level Grade Level of Complexity TotalArea Basic Proficient Advanced

Participatory 258 30.8%

131 15.6%

450 53.6%

839 100.0%

Mathematics Supported

Independent

220 26.5%

286 54.9%

341 41.0%

169 32.4%

270 32.5%

66 12.7%

831 100.0%

521 100.0%

3 Total

Participatory

764 34.9%

344 40.4%

641 29.3%

296 34.7%

786 35.9%

212 24.9%

2191 100.0%

852 100.0%

Reading Supported

Independent

117 24.3%

225 25.7%

211 43.8%

277 31.6%

154 32.0%

374 42.7%

482 100.0%

876 100.0%

Total 686 31.0%

784 35.5%

740 33.5%

2210 100.0%


137 17.4%

409 52.0%

787 100.0%


Independent

268 33.5%

275 42.3%

305 38.1%

282 43.4%

227 28.4%

93 14.3%

800 100.0%

650 100.0%

Total 784 35.0%

724 32.4%

729 32.6%

2237 100.0%


335 36.3%

270 29.3%

922 100.0%

4 Reading Supported

Independent

191 27.1%

227 36.5%

284 40.3%

204 32.8%

230 32.6%

191 30.7%

705 100.0%

622 100.0%

Total 735 32.7%

823 36.6%

691 30.7%

2249 100.0%


156 18.2%

473 55.3%

856 100.0%

Writing Supported

Independent

108 17.9%

251 33.0%

266 44.2%

279 36.7%

228 37.9%

231 30.4%

602 100.0%

761 100.0%

Total 586 26.4%

701 31.6%

932 42.0%

2219 100.0%


211 21.0%

507 50.3%

1007 100.0%

5 Mathematics Supported

Independent

226 33.2%

260 39.9%

269 39.5%

257 39.5%

186 27.3%

134 20.6%

681 100.0%

651 100.0%

Total 775 33.1%

737 31.5%

827 35.4%

2339 100.0% (cont‘d)


Grade Content Area Level of Complexity

BasicPerformance Level

Proficient Advanced Total


317 34.7%

267 29.2%

913 100.0%

Reading Supported

Independent

190 30.4%

240 29.2%

251 40.1%

332 40.4%

185 29.6%

249 30.3%

626 100.0%

821 100.0%

5 Total

Participatory

759 32.2%

291 43.6%

900 38.1%

199 29.8%

701 29.7%

177 26.5%

2360 100.0%

667 100.0%

Science Supported

Independent

140 32.0%

305 25.2%

161 36.8%

400 33.1%

136 31.1%

504 41.7%

437 100.0%

1209 100.0%

Total 736 31.8%

760 32.9%

817 35.3%

2313 100.0%


219 26.1%

348 41.5%

838 100.0%


Independent

166 23.8%

286 38.0%

311 44.6%

300 39.8%

220 31.6%

167 22.2%

697 100.0%

753 100.0%

6 Total

Participatory

723 31.6%

273 37.6%

830 36.3%

237 32.6%

735 32.1%

217 29.8%

2288 100.0%

727 100.0%

Reading Supported

Independent

134 23.4%

345 35.1%

227 39.6%

303 30.9%

212 37.0%

334 34.0%

573 100.0%

982 100.0%

Total 752 33.0%

767 33.6%

763 33.4%

2282 100.0%


164 20.0%

348 42.5%

819 100.0%


Independent

234 27.2%

336 43.6%

342 39.8%

265 34.4%

283 32.9%

169 21.9%

859 100.0%

770 100.0%

7 Total

Participatory

877 35.8%

289 41.7%

771 31.5%

217 31.3%

800 32.7%

187 27.0%

2448 100.0%

693 100.0%

Reading Supported

Independent

133 28.2%

329 25.4%

156 33.1%

500 38.6%

182 38.6%

468 36.1%

471 100.0%

1297 100.0%

Total 751 30.5%

873 35.5%

837 34.0%

2461 100.0% (cont‘d)






229 29.9%

220 28.8%

765 100.0%


Independent

280 26.3%

314 47.9%

466 43.7%

219 33.4%

320 30.0%

122 18.6%

1066 100.0%

655 100.0%

Total 910 36.6%

914 36.8%

662 26.6%

2486 100.0%


241 30.3%

259 32.5%

796 100.0%

Reading Supported

Independent

187 28.1%

365 35.6%

231 34.7%

350 34.1%

248 37.2%

310 30.2%

666 100.0%

1025 100.0%

8 Total

Participatory

848 34.1%

260 29.9%

822 33.1%

241 27.7%

817 32.9%

369 42.4%

2487 100.0%

870 100.0%

Writing Supported

Independent

140 24.9%

277 28.3%

229 40.7%

331 33.8%

194 34.5%

370 37.8%

563 100.0%

978 100.0%

Total 677 28.1%

801 33.2%

933 38.7%

2411 100.0%


193 30.8%

165 26.3%

627 100.0%

Science Supported

Independent

118 28.4%

294 20.8%

156 37.6%

345 24.4%

141 34.0%

773 54.7%

415 100.0%

1412 100.0%

Total 681 27.8%

694 28.3%

1079 44.0%

2454 100.0%


312 35.8%

237 27.2%

872 100.0%


Independent

256 28.0%

380 32.9%

389 42.6%

397 34.4%

269 29.4%

378 32.7%

914 100.0%

1155 100.0%

9 Total

Participatory

959 32.6%

295 37.3%

1098 37.3%

227 28.7%

884 30.1%

268 33.9%

2941 100.0%

790 100.0%

Reading Supported

Independent

166 28.5%

426 27.4%

182 31.2%

415 26.7%

235 40.3%

713 45.9%

583 100.0%

1554 100.0%

Total 887 30.3%

824 28.2%

1216 41.5%

2927 100.0% (cont‘d)






336 39.6%

210 24.8%

848 100.0%


Independent

248 22.0%

387 36.5%

546 48.4%

464 43.8%

334 29.6%

209 19.7%

1128 100.0%

1060 100.0%

Total 937 30.9%

1346 44.3%

753 24.8%

3036 100.0%


228 28.7%

286 36.0%

794 100.0%

10 Reading Supported

Independent

197 25.5%

490 33.5%

291 37.6%

587 40.2%

285 36.9%

384 26.3%

773 100.0%

1461 100.0%

Total 967 31.9%

1106 36.5%

955 31.5%

3028 100.0%


283 28.6%

474 47.9%

990 100.0%

Writing Supported

Independent

228 26.8%

431 37.7%

343 40.3%

310 27.1%

280 32.9%

403 35.2%

851 100.0%

1144 100.0%

Total 892 29.9%

936 31.4%

1157 38.8%

2985 100.0%


181 27.8%

212 32.6%

650 100.0%

11 Science Supported

Independent

135 22.6%

442 25.0%

280 46.8%

575 32.5%

183 30.6%

752 42.5%

598 100.0%

1769 100.0%

Total 834 27.6%

1036 34.3%

1147 38.0%

3017 100.0%


8.2 Item Statistics

Highlights:

Classical Item Statistics Across Grade Levels ¢ Item Difficulty ¢ Item Discrimination

£ Mean score / Total £ Pearson product number of points moment correlation

£ Typical £ Typical ¢ 0.25 œ 0.90 ¢ 0.2 œ 0.6 ¢ Mathematics ¢ Mathematics

£ 0.28 œ 0.74 £ 0.48 œ 0.82 ¢ Reading ¢ Reading

£ 0.39 œ 0.73 £ 0.55 œ 0.86 ¢ Science ¢ Science

£ 0.33 œ 0.77 £ 0.52 œ 0.89 ¢ Writing ¢ Writing ¢ 0.44 œ 0.75 £ 0.61 œ 0.87

As noted in Brown (1983), —A test is only as good as the items it contains.“ A complete

evaluation of a test‘s quality must include an evaluation of each item. Both the Standards for

Educational and Psychological Testing (AERA, APA, NCME, 1999) and the Code of Fair Testing

Practices in Education (JCTP, 1988) include standards for identifying quality items. Test items

should assess only knowledge or skills that are identified as part of the domain being tested and

should avoid assessing irrelevant factors. They should also be unambiguous and free of grammatical

errors, potentially insensitive content or language, and other confounding characteristics. Further,

items must not disadvantage test takers from particular racial, ethnic, or gender groups.

Both qualitative and quantitative analyses were conducted to ensure that the Florida Alternate

Assessment items met these standards. Earlier chapters of this report discussed qualitative checks on

item quality. The following discussion focuses on three categories of quantitative evaluation of

2007œ08 Florida Alternate Assessment items: (1) difficulty indices, (2) discrimination indices, and

(3) subgroup differences in item performance (i.e., differential item functioning).


As explained earlier in this report, each grade-content test of the Florida Alternate

Assessment was composed exclusively of selected-response items (at the Participatory level, all

items were multiple-choice with 3 response options; i.e., the key and two distracters). Sixteen —sets“

of three MC items comprised a test: one item each at the Participatory, Supported, and Independent

Levels of Complexity.

The Participatory level items were administered up to three times to a student, depending on

whether the student responded correctly on the first try or needed scaffolding once or twice. (The

first scaffold removed a distracter, leaving the key and one distracter; the second scaffold removed

the remaining distracter, leaving only the key.) According to the scoring rubric, 3, 2, or 1 point was

awarded to the student, respectively, under these contingencies.

If and only if 3 points were earned on a Participatory item, the student was administered the

Supported level item one time. The student either answered correctly, earning an additional 3 points

for the set, or incorrectly, ending with a final item-set score of 3 points. If and only if 6 points were

earned on the Supported level item, the student was given a single opportunity on the Independent

level, and ended with an item-set score of 9 points for responding correctly or 6 points for

responding incorrectly.

In summary, Florida Alternate Assessment grade-content tests can be thought of as 16-item

tests if the Participatory, Supported, and Independent items are considered in sets. The scoring rubric

does just that, and treats each set as a polytomous item with six possible item scores: 0, 1, 2, 3, 6, 9.

The maximum possible total raw score is 144. Item statistics for the grade-content tests are described

and tabulated in the following subsections.

8.2.1 Difficulty Indices

All Florida Alternate Assessment items were evaluated in terms of difficulty according to

standard classical test theory (CTT) practice. The expected item difficulty, also known as the p-


value, is the main index of item difficulty under the CTT framework. This index measures an item‘s

difficulty by averaging the proportion of points received across all students who took the item. In

order to place all items on a 0œ1 scale, the p-value was computed as the average score on the item

divided by its maximum possible score. Although the p-value is traditionally called a measure of

difficulty, it is properly interpreted as an easiness index, because larger p-values indicate easier

items. An index of 0.00 indicates that no student received credit for the item. At the opposite

extreme, an index of 1.00 indicates that every student received full credit for the item.

Items that are answered correctly by almost all students provide little information about

differences in student ability, but they do indicate knowledge or skills that have been mastered by

most students. The converse is true of items that are incorrectly answered by most students. In

general, to provide the most precise measurement, difficulty indices should range from near-chance

performance (0.25 for four-option MC items, 0.00 for CR items) to 0.90. Experience has indicated

that items conforming to this guideline provide satisfactory statistical information for the bulk of the

student population. However, on a criterion-referenced test such as Florida‘s Alternate Assessment,

it may be appropriate to include some items with difficulty values outside this region in order to

measure the skill that exists at a given grade level throughout the range. A generous range of item

difficulties also helps insure against excesses of scores at the floor or ceiling of the distribution.

8.2.2 Item Discrimination

It is a desirable feature of an item when higher-ability students perform better on it than do

lower-ability students. A commonly used measure of this characteristic is the correlation between

student performance on the item and total test score. Within CTT, this item-test correlation is

referred to as the item‘s discrimination, because, in effect, the strength of the correlation indicates

the extent to which successful performance on an item discriminates between high and low scores on

the test. For items on the 2007œ08 Florida Alternate Assessment, the Pearson product-moment


correlation was used as the item discrimination index. The theoretical range of these statistics is œ1.0

to +1.0, with a typical range from +0.2 to +0.6.

One can think of a discrimination index as a measure of how closely an item assesses the

same knowledge and skills as other items that contribute to the criterion total score, in other words,

as a measure of construct consistency. It is therefore quite important to select an appropriate total

score criterion; for the 2007œ08 Florida Alternate Assessment, total raw score was used. Item-test

correlations were computed for each item, and results are summarized in the next section.

8.2.3 Summary of Item Analysis Results

Appendix I presents p-values and discriminations calculated on four bases: (1) using all

students, (2) using participatory students, (3) using supported students, and (3) using independent

students.

In general, when calculated for all students, the item difficulty and discrimination indices

were in acceptable and expected ranges. Across all grades and content areas, p-values fell between

0.28 and 0.77. Discrimination indices ranged between 0.48 and 0.89. Positive discrimination indices

indicate that students who performed well on individual items tended to perform well overall, and

vice-versa. As might be expected, the items appear easier (higher p-values) when calculated using

the independent students, slightly more difficult (i.e., lower p-values) when calculated using the

supported students, and still more difficult when calculated using the participatory students. The

items are most discriminating for the participatory students, slightly less discriminating for the

independent students, and not discriminating for the supported students. This trend in discrimination

indices occurs at all grade-contents and is an artifact of the scale used to score the items and of the

restriction of range that results from the scale and splitting the students according to three reduced

score ranges, which limits the possible variation of the scores and in turn the range of possible item

discriminations.


Very few items were answered correctly at near-chance or near-perfect rates. The standard

deviations of the item scores are somewhat larger than might typically be expected, an artifact again

of the scale used to score the items.

To compare indices such as these, that are population dependent, across grade and content

areas, is complicated. Direct comparisons would require that either items or students were common

across groups, and here they were not.

8.3 Differential Item Functioning

Highlights:

DIF ¢ Mantel-Haenszel & Standardized P

£Negligible DIF ¢ -0.05 to 0.05

£Low DIF ¢ -0.10 to -0.05 or 0.05 to 0.10

£High DIF ¢ < - 0.10 or >0.10

¢ Male/Female, White/Black, & White/Hispanic £All Items classified as having Negligible DIF

The Code of Fair Testing Practices in Education (Joint Committee on Testing Practices,

1988) explicitly states that subgroup differences in performance should be examined when sample

sizes permit, and actions should be taken to make certain that differences in performance are due to

construct-relevant, rather than construct-irrelevant, factors. The Standards for Educational and

Psychological Testing (AERA, APA, NCME, 1999) includes similar guidelines. As part of the effort

to identify such problems, Florida Alternate Assessment items were evaluated by means of

differential item functioning (DIF) statistics.


DIF procedures are designed to identify items on which the performances of certain

subgroups of interest differ from each other after controlling for construct-relevant achievement. In

order to ensure meaningful results, DIF statistics were not computed unless there were at least 200

students in both subgroups. As a result, DIF comparisons were conducted between male and female,

White and Hispanic, and White and Black for all grade-content combinations.

A two-step process utilizing both the Mantel-Haenszel (Holland and Thayer, 1988) and

standardization (Dorans and Kulick, 1986) procedures was employed to detect DIF. Both of these

procedures calculate the difference in item performance for groups of students matched for

achievement on the total test. In the first step, the Mantel-Haenszel procedure was used to identify

items that showed statistically significant DIF.

2

∑Rrm − ∑ ( rm )E R − .5 MHχ 2 = m m ,

∑ ( )Var R rm m

where, r represents the reference group, R represents the number of students who got the item correct, and m represents each score level.

The expected proportion correct is represented by

rm rm∑ ( rm ) N R ,E R =

m Ntm

where, t represents the total group and N represent the number of students who took the item.

The variance of Rrm is calculated as follows:

N R[ rm tm N fmWrm ]( rm ) = 2Var R [N N( −1)] tm tm

where, f represents the focal group, W represents the number of students who got the item wrong.

Because of the large number of students on which the calculations were based, the majority

of the items indicated a statistically significant difference between the focal and reference groups.


(Note that this issue is not specific to Mantel-Haenszel calculations. Large enough sample sizes will

indicate that even trivially small results are statistically significant.) For this reason, in the second

step of the process, the standardization procedure was used to categorize items according to the

amount of DIF detected.

The standardization DIF procedure (Dorans & Kulick, 1986) calculates the difference in item

performance for two groups of students (at a time) matched for achievement on the total test.

∑w Pm ( fm − Prm ) mSTD P DIF = ,∑wm

m

where, f represents the focal group, r represents the reference group, P represents the proportion correct, m represents each score level, and w represents a weighting factor to account for the different number of students at each score level.

Specifically, average item performance is calculated for students at every total score. Then an

overall average is calculated, weighting the total score distribution so that it is the same for the two

groups. The criterion for the 2007œ08 Florida Alternate Assessment was total score. It should be

noted that differential performances between groups may or may not be indicative of bias in the test.

Group differences in course-taking patterns, interests, or school curricula can lead to DIF as well. If

subgroup differences are related to construct-relevant factors, items should be considered for

inclusion on a test.

Computed DIF indices have a theoretical range from œ1.00 to 1.00 for MC items; those for

polytomous items are adjusted to the same scale. For reporting purposes, items were categorized

according to DIF index range guidelines suggested by Dorans and Holland (1993). Indices between

œ0.05 and 0.05 (Type A) can be considered —negligible.“ Most items should fall in this range. DIF

indices between œ0.10 and œ0.05 or between 0.05 and 0.10 (Type B) can be considered —low DIF“

but should be inspected to ensure that no possible effect is overlooked. Items with DIF indices


outside the [œ0.10, 0.10] range (Type C) can be considered —high DIF“ and should trigger careful

examination.

Tables 9-3 through 9-5 show the number of items classified into each DIF category, broken

down by grade and content area. Respectively, the comparisons shown are between male and female,

White and Black, and White and Hispanic.

Table 8-3. 2007–08 Florida Alternate Assessment: Number of Items Classified into Differential Item Functioning

(DIF) Categories* by Grade and Content—Male/Female Grade Content Area A B C D

3 Math Reading

16 16

0 0

0 0

0 0

Math 16 0 0 0 4 Reading 16 0 0 0

Writing 16 0 0 0 Math 16 0 0 0

5 Reading 16 0 0 0 Science 16 0 0 0

6 Math Reading

16 16

0 0

0 0

0 0

7 Math Reading

16 16

0 0

0 0

0 0

Math 16 0 0 0

8 Reading Science

16 16

0 0

0 0

0 0

Writing 16 0 0 0

9 Math Reading

16 16

0 0

0 0

0 0

Math 16 0 0 0 10 Reading 16 0 0 0

Writing 16 0 0 0 11 Science 16 0 0 0

A=negligible; B=low; C=high; D=not enough students in subgroup(s) to compute DIF



(DIF) Categories* by Grade and Content—White/Black Grade Content Area A B C D

3 Math Reading

16 16

0 0

0 0

0 0

Math 16 0 0 0 4 Reading 16 0 0 0

Writing 16 0 0 0 Math 16 0 0 0


6 Math Reading

16 16

0 0

0 0

0 0

7 Math Reading

16 16

0 0

0 0

0 0

Math 16 0 0 0

8 Reading Science

16 16

0 0

0 0

0 0

Writing 16 0 0 0

9 Math Reading

16 16

0 0

0 0

0 0

Math 16 0 0 0 10 Reading 16 0 0 0

Writing 16 0 0 0 11 Science 16 0 0 0

* A=negligible; B=low; C=high; D=not enough students in subgroup(s) to compute DIF


(DIF) Categories* by Grade and Content— White/Hispanic Grade Content Area A B C D

3 Math Reading

16 16

0 0

0 0

0 0

Math 16 0 0 0 4 Reading 16 0 0 0

Writing 16 0 0 0 Math 16 0 0 0


6 Math Reading

16 16

0 0

0 0

0 0

7 Math Reading

16 16

0 0

0 0

0 0

Math 16 0 0 0

8 Reading Science

16 16

0 0

0 0

0 0

Writing 16 0 0 0

9 Math Reading

16 16

0 0

0 0

0 0

Math 16 0 0 0 10 Reading 16 0 0 0

Writing 16 0 0 0 11 Science 16 0 0 0

* A=negligible; B=low; C=high; D=not enough students in subgroup(s) to compute DIF


The tables show that all DIF distinctions in the 2007œ08 Florida Alternate Assessment tests

were —Type A,“ i.e., —negligible“ DIF (Dorans and Holland 1993). Both the Code of Fair Testing

Practices in Education (Joint Committee on Testing Practices, 1988) and the Standards for

Educational and Psychological Testing (AERA, 1999) assert that test items must be free from

construct-irrelevant sources of differential difficulty. If subgroup differences in performance can be

plausibly attributed to construct-relevant factors, the items may be included on a test. What is

important is to determine whether the cause of this differential performance is construct-relevant.

Table 8-6 presents the number of items classified into each DIF category by direction for the

male versus female comparison. For example, the —Favor Female“ column under —Negligible DIF

(A)“ denotes the total number of items classified in category A on which females performed better

than males relative to performance on the test as a whole. —Favor Male“ is shown in the next column.

The —N“ and —%“ columns display the aggregate number and proportion of items in the category,

respectively. Results are broken out by grade and content area.


Table 8-6. 2007–08 Florida Alternate Assessment: Number and Proportion of Items Classified into Each DIF Category, with Advantage Indicated—Male versus Female

Negligible DIF (A) Low DIF (B) High DIF (C) Content Grade Favor Favor Favor Favor Favor Favor Area N % N % N %Female Male Female Male Female Male Math 7 9 16 100 0 0 0 0 0 0 0 03 Reading 9 7 16 100 0 0 0 0 0 0 0 0 Math 9 7 16 100 0 0 0 0 0 0 0 0

4 Reading 11 5 16 100 0 0 0 0 0 0 0 0 Writing 12 4 16 100 0 0 0 0 0 0 0 0 Math 13 3 16 100 0 0 0 0 0 0 0 0

5 Reading 10 6 16 100 0 0 0 0 0 0 0 0 Science 10 6 16 100 0 0 0 0 0 0 0 0

6 Math Reading

9 11

7 5

16 16

100 100

0 0

0 0

0 0

0 0

0 0

0 0

0 0

0 0

7 Math Reading

9 11

7 5

16 16

100 100

0 0

0 0

0 0

0 0

0 0

0 0

0 0

0 0

Math 11 5 16 100 0 0 0 0 0 0 0 0

8 Reading Science

11 10

5 6

16 16

100 100

0 0

0 0

0 0

0 0

0 0

0 0

0 0

0 0

Writing 12 4 16 100 0 0 0 0 0 0 0 0

9 Math Reading

8 8

8 8

16 16

100 100

0 0

0 0

0 0

0 0

0 0

0 0

0 0

0 0

Math 9 7 16 100 0 0 0 0 0 0 0 0 10 Reading 12 4 16 100 0 0 0 0 0 0 0 0

Writing 11 5 16 100 0 0 0 0 0 0 0 0 11 Science 9 7 16 100 0 0 0 0 0 0 0 0

—Favor“ indicates which gender was advantaged in the category (controlling for total test score).


Chapter 9. RELIABILITY

Highlights:

Test Reliability Across Grade Levels

¢ Cronbach‘s Alpha ¢ SEM (Classical) £ Mathematics £ Mathematics

¢ 0.94 to 0.96 ¢ 7.32 to 8.82

£ Reading £ Reading ¢ 0.95 to 0.97 ¢ 7.12 to 8.14

£ Science £ Science ¢ 0.96 to 0.97 ¢ 7.38 to 8.08

£ Writing £ Writing ¢ 0.96 to 0.97 ¢ 6.69 to 7.29

In an earlier section, individual item characteristics of the 2007-08 Florida Alternate

Assessment were presented. Although individual item performance is an important focus for

evaluation, a complete evaluation of an assessment must also address the way in which items

function together and complement one another. Any measurement includes some amount of

measurement error. No academic assessment can measure student performance with perfect

accuracy; some students will receive scores that underestimate their true ability, and other students

will receive scores that overestimate their true ability. Items that function well together produce

assessments that have less measurement error (i.e., the error is small on average). Such assessments

are described as —reliable.“

There are a number of ways to estimate an assessment‘s reliability. One approach is to split

all test items into two groups and then correlate students‘ scores on the two half-tests. This is known

as a split-half estimate of reliability. If the two half-test scores correlate highly, the items on them

are likely measuring very similar knowledge or skills. It suggests that measurement error will be

minimal.

Chapter 9—Reliability 73 2007–08 Florida Alt Technical Report

The split-half method requires psychometricians to select items that contribute to each half-

test score. This decision may have an impact on the resulting correlation, since each different

possible split of the test halves will result in a different correlation. Another problem with the split-

half method of calculating reliability is that it underestimates reliability, because test length is cut in

half. All else being equal, a shorter test is less reliable than a longer test. Cronbach (1951) provided a

statistic, alpha (α), which avoids these concerns of the split-half method by comparing individual

item variances to total test variance. Cronbach‘s α was used to assess the reliability of the 2007œ08

Florida Alternate Assessment tests. The formula is as follows:

n 2

∑σ ( )n Yi i=1α ≡ 1− 2

n −1 σ x

Where: I indexes the item, n is the number of items,

2σ ( ) represents individual item variance, and Yi

2σ represents the total test variance x

Table 9-1 presents descriptive statistics, Cronbach‘s α coefficient, and raw score standard

errors of measurement (SEMs) for each content area and grade.


Table 9-1. 2007–08 Florida Alternate Assessment: Raw Score Descriptive Statistics, Reliabilities, and Standard

Errors of Measurement by Grade and Content Area (All Students) Grade Content Area N Possible

Score Min

Score Max

Score Mean Score

Score SD

Reliability (α) S.E.M.

3 Mathematics Reading

2191 2210

144 144

0 0

144 144

68.992 77.672

36.016 40.734

0.94 0.96

8.822 8.147

Mathematics 2237 144 0 144 72.734 36.599 0.96 7.320 4 Reading 2249 144 0 144 71.916 36.340 0.95 8.126

Writing 2219 144 0 144 76.608 38.595 0.97 6.685 Mathematics 2339 144 0 144 69.802 38.075 0.96 7.615

5 Reading 2360 144 0 144 75.743 38.756 0.96 7.751 Science 2313 144 0 144 88.821 42.600 0.97 7.379


2288 2282

144 144

0 0

144 144

74.487 82.486

38.467 38.886

0.96 0.96

7.693 7.777


2448 2461

144 144

0 0

144 144

75.188 88.828

36.846 41.210

0.95 0.97

8.239 7.138

Mathematics 2486 144 0 144 73.272 34.802 0.95 7.782

8 Reading Science

2487 2454

144 144

0 0

144 144

81.912 94.872

39.059 43.493

0.96 0.97

7.812 7.533

Writing 2411 144 0 144 80.829 40.637 0.97 7.039


2941 2927

144 144

0 0

144 144

81.141 91.704

38.564 41.112

0.95 0.97

8.623 7.121


Writing 2985 144 0 144 81.543 36.462 0.96 7.292 11 Science 3017 144 0 144 96.491 40.408 0.96 8.082

The large spread of scores is an artifact of the rubric used to score the assessment. Because

each item is scored 0, 1, 2, 3, 6, or 9, the standard deviation of the total test score across students

appears larger that what might be considered typical. For reading the reliability coefficient ranged

from 0.95 to 0.97, for mathematics from 0.94 to 0.96, for writing from 0.96 to 0.97, and for science

from 0.96 to 0.97. Although these reliabilities are quite high, they are somewhat typical of alternate

assessments constructed of polytomous scored items. These statistics were also calculated separated

for participatory students, supported students, and independent students and presented in Tables 10-2

through 10-4.The reliability coefficients calculated on just the participatory students were high, those

on the independent students somewhat lower, and those using the supported students even lower.


This trend is consistent across all grade content areas and is an artifact of the scoring rubric and

restriction of range in student abilities.


Errors of Measurement by Grade and Content Area (Participatory Students) Grade Content Area N Possible

Score Min

Score Max

Score Mean Score

Score SD



2191 2210

144 144

0 0

57 62

30.45 32.86

16.65 17.59

0.88 0.90

5.77 5.56



5 Reading 2360 144 0 62 34.06 17.06 0.90 5.40 Science 2313 144 0 61 30.93 17.67 0.91 5.30


2288 2282

144 144

0 0

57 62

31.57 33.72

16.24 18.14

0.88 0.90

5.63 5.74


2448 2461

144 144

0 0

57 62

32.33 31.63

17.10 17.65

0.91 0.91

5.13 5.29

Mathematics 2486 144 0 57 30.63 16.71 0.92 4.73

8 Reading Science

2487 2454

144 144

0 0

62 61

33.98 30.55

18.04 17.57

0.90 0.89

5.71 5.83

Writing 2411 144 0 63 34.97 17.95 0.93 4.75


2941 2927

144 144

0 0

57 62

32.58 34.31

16.33 17.97

0.88 0.90

5.66 5.68


Writing 2985 144 0 63 38.65 17.62 0.93 4.66 11 Science 3017 144 0 61 31.76 18.50 0.90 5.85



Errors of Measurement by Grade and Content Area (Supported Students) Grade Content Area N Possible

Score Min

Score Max

Score Mean Score

Score SD



2191 2210

144 144

58 63

98 98

79.13 81.54

11.64 10.03

0.27 0.11

9.94 9.47



5 Reading 2360 144 63 98 80.69 10.34 0.16 9.48 Science 2313 144 62 98 80.21 10.90 0.25 9.44


2288 2282

144 144

58 63

98 98

79.36 82.36

11.43 10.01

0.30 0.11

9.56 9.45


2448 2461

144 144

58 63

98 98

78.83 81.64

11.99 10.45

0.37 0.15

9.52 9.63

Mathematics 2486 144 58 98 78.15 11.27 0.40 8.73

8 Reading Science

2487 2454

144 144

63 62

98 98

81.16 80.99

10.17 10.89

0.19 0.16

9.16 9.98

Writing 2411 144 64 98 80.41 10.09 0.38 7.94


2941 2927

144 144

58 63

98 98

78.30 81.36

11.71 10.63

0.29 0.17

9.86 9.69


Writing 2985 144 64 98 81.62 9.86 0.31 8.19 11 Science 3017 144 62 98 81.44 10.65 0.17 9.70



Errors of Measurement by Grade and Content Area (Independent Students) Grade Content Area N Possible

Score Min

Score Max

Score Mean Score

Score SD



2191 2210

144 144

99 99

144 144

114.90 119.13

11.30 12.16

0.45 0.55

8.38 8.16



5 Reading 2360 144 99 144 123.87 12.30 0.66 7.17 Science 2313 144 99 144 117.73 11.79 0.50 8.34


2288 2282

144 144

99 99

144 144

118.66 116.70

11.90 12.34

0.55 0.58

7.98 8.00


2448 2461

144 144

99 99

144 144

122.00 115.14

12.19 12.16

0.62 0.58

7.51 7.88

Mathematics 2486 144 99 144 119.63 13.17 0.68 7.45

8 Reading Science

2487 2454

144 144

99 99

144 144

127.52 121.86

12.45 13.05

0.69 0.72

6.93 6.91

Writing 2411 144 99 144 120.06 13.21 0.62 8.14


2941 2927

144 144

99 99

144 144

124.76 116.20

13.00 11.95

0.67 0.57

7.47 7.83


Writing 2985 144 99 144 125.37 13.04 0.69 7.26 11 Science 3017 144 99 144 114.90 11.30 0.45 8.38

Because different grades and content areas have different access points and benchmarks, it is

inappropriate to make inferences about the quality of one test by comparing its reliability to that of

another test from a different grade and/or content area.


9.1 Reliability of Performance Level Categorization

Highlights:

Classification ¢ Accuracy ¢ Consistency

£ Match between actual £ Match between actual decisions and error-free decisions and those from a decisions parallel form

¢ Mathematics ¢ Mathematics £ Overall: 0.76 - 0.79 £ Overall: 0.68 - 0.72 £ Cut score: > 0.90 £ Cut score: > 0.86

¢ Reading ¢ Reading £ Overall: 0.76 - 0.78 £ Overall: 0.67 - 0.71 £ Cut score: > 0.89 £ Cut score: > 0.85

¢ Science ¢ Science £ Overall: 0.79 - 0.80 £ Overall: 0.71 - 0.73 £ Cut score: > 0.91 £ Cut score: > 0.86

¢ Writing ¢ Writing £ Overall: 0.79 - 0.80 £ Overall: 0.72 - 0.73 £ Cut score: > 0.91 £ Cut score: > 0.91

All test scores contain measurement error; thus, classifications based on test scores are also

subject to measurement error. Empirical analyses were conducted to determine the statistical

accuracy and consistency of performance level classifications on the 2007œ08 Florida Alternate

Assessment.

It must be noted before proceeding that three cut scores typically divide the entire score

range into four or five performance levels on general assessments. On the Florida Alternate

Assessment 16-item grade-content tests, however, (three) performance levels are nested within each

level of complexity that divide the raw score range. One thus encounters the unusual situation that,

for example, a student classified as Advanced within the Participatory level of complexity has a

substantially lower raw score than does a student classified as Basic within the Independent level of

complexity.

Psychometricians at Measured Progress determined that three cut points divide each Florida

Alternate Assessment grade-content test score range into four critical classifications from the

standpoint of accountability: the cuts that differentiate between non-proficiency and proficiency at


each level of complexity (i.e., between Participatory-Basic and Participatory-Proficient, between

Supported-Basic and Supported-Proficient, and between Independent-Basic and Independent-

Proficient). The three cut points thereby result in four classifications: (1) below Participatory-

Proficient, (2) Participatory-Proficient through Supported-Basic, (3) Supported-Proficient through

Independent-Basic, and (4) Independent-Proficient and above. It is the accuracy and consistency of

these classifications that are described below.

9.1.1 Computing Accuracy and Consistency

Accuracy refers to the extent to which decisions based on test scores match decisions that

would have been made if the scores did not contain any measurement error. Accuracy must be

estimated, because errorless test scores do not exist. Consistency measures the extent to which

classification decisions based on test scores match the decisions based on scores from a second,

parallel form of the same test. Consistency can be evaluated directly from actual responses to test

items if two complete and parallel forms of the test are given to the same group of students. In

operational assessment programs, however, such a design is usually impractical. Instead, techniques,

such as those due to Livingston and Lewis (1995), have been developed to estimate both the

accuracy and consistency of classification decisions based on a single administration of a test. The

Livingston and Lewis technique was used for the 2007œ08 Florida Alternate Assessment because it

is easily adaptable to examinations of all kinds of formats, including mixed-format tests.

The accuracy and consistency estimates reported here make use of —true scores“ in the

classical test theory sense. A true score is the score that would be obtained if a test had no

measurement error. Of course, true scores cannot be observed and so must be estimated. In the

Livingston and Lewis method, estimated true scores are used to classify students into their —true“

classification.


For the 2007œ08 Florida Alternate Assessment, after various technical adjustments were

made (described in Livingston and Lewis, 1995), a 4 × 4 contingency table of accuracy was created

for each content area and grade, where cell [I,j] represented the estimated proportion of students

whose true score fell into classification I (where I = 1 to 4) and observed score into classification j

(where j = 1 to 4). The sum of the diagonal entries, i.e., the proportion of students whose true and

observed classifications matched one another, signified overall accuracy.

To estimate consistency, true scores were used to estimate the joint distribution of classifica-

tions on two independent, parallel test forms. Following statistical adjustments (per Livingston and

Lewis, 1995), a new 4 × 4 contingency table was created for each content area and grade and

populated by the proportion of students who would be classified into each combination of

classifications according to the two (hypothetical) parallel test forms. Cell [I,j] of this table

represented the estimated proportion of students whose observed score on the first form would fall

into classification I (where I = 1 to 4) and whose observed score on the second form would fall into

classification j (where j = 1 to 4). The sum of the diagonal entries, i.e., the proportion of students

classified by the two forms into exactly the same classification, signified overall consistency.

Another way to measure consistency is to use Cohen‘s (1960) coefficient κ (kappa), which

assesses the proportion of consistent classifications after removing the proportion of consistent

classifications that would be expected by chance. It is calculated using the following formula

∑Cii − ∑C C i. .i(Observed agreement) - (Chance agreement) i iκ = = ,

1− ∑ i. .i1 - (Chance agreement) C C i

where: Ci. is the proportion of students whose observed classification would be Level I (where I = 1 œ 4) on the first hypothetical parallel form of the test; C.i is the proportion of students whose observed classification would be Level I (where I = 1 œ 4) on the second hypothetical parallel form of the test; Cii is the proportion of students whose observed classification would be Level I (where I = 1 œ 4) on both hypothetical parallel forms of the test.

Because κ is corrected for chance, its values are lower than are other consistency estimates.


9.1.2 Accuracy and Consistency Results

Accuracy and consistency analyses overall, conditional on classification, and at three cut

scores are summarized in Table 9-2, while more detailed accuracy and consistency results may be

found in Appendix J.

For accuracy and consistency conditional upon classification, the denominator is the

proportion of students associated with a given classification. For example, the conditional accuracy

value is 0.80 for the second classification (i.e., that below Supported-Proficient and above

Participatory-Proficient) for mathematics grade 3. This figure indicates that among the students

whose true scores placed them in this second classification, 80% of them would be expected to be in

this classification when categorized according to their observed score. Similarly, the corresponding

consistency value of 0.75 indicates that 75% of students with observed scores in the second

classification would be expected to score in this classification again if a second, parallel test form

were used.

False positive and false negative decision rates are reported with the accuracy and

consistency estimates at the three cuts. False positives are the proportion of students whose observed

scores were above the cut and true scores below the cut. False negatives are the proportion of

students whose observed scores were below the cut and true scores above the cut.

It should be noted that Livingston & Lewis (1995) discuss two versions of the accuracy and

consistency tables. A standard version performs calculations for forms parallel to the form taken. An

—adjusted“ version adjusts the results of one form to match the observed score distribution obtained

in the data. The calculations described here use the standard version for two reasons: 1) this

—unadjusted“ version can be considered a smoothing of the data, thereby decreasing the variability of

the results; and 2) for results dealing with the consistency of two parallel forms, the unadjusted

tables are symmetric, indicating that the two parallel forms have the same statistical properties. This


second reason is consistent with the notion of forms that are parallel, i.e., it is more intuitive and

interpretable for two parallel forms to have the same statistical distribution as one another.

Table 9-2 summarizes most of the results of the classification tables at a glance. As with

other types of reliability, it is inappropriate when analyzing the decision accuracy and consistency of

a given examination to compare results between grades and content areas.


Table 9-2. 2007–08 Florida Alternate Assessment: Summary of

Decision Accuracy (and Consistency) Results for Proficiency Cuts on the Raw Score Scale

Content/ Conditional on Level At Basic-Proficient Cut Point within LOC Grade Overall Below P-P P-P thru S-B S-P thru I-B I-P and above Participatory Supported Independent

Math/3 0.79 (0.71) 0.72 (0.60) 0.80 (0.75) 0.76 (0.68) 0.86 (0.70) 0.96 (0.94) 0.90 (0.86) 0.94 (0.91) Math/4 0.79 (0.72) 0.74 (0.62) 0.80 (0.75) 0.76 (0.69) 0.88 (0.74) 0.96 (0.94) 0.90 (0.87) 0.93 (0.90) Math/5 0.78 (0.69) 0.68 (0.57) 0.79 (0.73) 0.75 (0.66) 0.88 (0.75) 0.93 (0.91) 0.91 (0.87) 0.93 (0.91) Math/6 0.76 (0.68) 0.75 (0.66) 0.75 (0.68) 0.72 (0.64) 0.87 (0.74) 0.95 (0.93) 0.91 (0.87) 0.91 (0.88) Math/7 0.77 (0.69) 0.75 (0.64) 0.76 (0.69) 0.75 (0.67) 0.88 (0.75) 0.95 (0.93) 0.90 (0.86) 0.92 (0.89) Math/8 0.78 (0.70) 0.76 (0.65) 0.76 (0.70) 0.76 (0.69) 0.87 (0.73) 0.95 (0.93) 0.90 (0.86) 0.93 (0.90) Math/9 0.76 (0.68) 0.75 (0.66) 0.71 (0.62) 0.71 (0.63) 0.90 (0.78) 0.95 (0.93) 0.91 (0.87) 0.91 (0.87) Math/10 0.77 (0.69) 0.75 (0.64) 0.73 (0.65) 0.75 (0.68) 0.88 (0.76) 0.96 (0.94) 0.90 (0.87) 0.91 (0.88) Reading/3 0.76 (0.68) 0.76 (0.69) 0.74 (0.65) 0.65 (0.55) 0.89 (0.80) 0.94 (0.91) 0.92 (0.89) 0.90 (0.87) Reading/4 0.76 (0.68) 0.76 (0.67) 0.77 (0.71) 0.70 (0.61) 0.88 (0.74) 0.94 (0.92) 0.90 (0.86) 0.92 (0.89) Reading/5 0.76 (0.67) 0.75 (0.67) 0.76 (0.68) 0.68 (0.59) 0.87 (0.76) 0.94 (0.92) 0.91 (0.88) 0.91 (0.87) Reading/6 0.76 (0.67) 0.78 (0.70) 0.73 (0.65) 0.67 (0.57) 0.87 (0.77) 0.95 (0.94) 0.92 (0.88) 0.89 (0.85) Reading/7 0.77 (0.69) 0.72 (0.65) 0.71 (0.61) 0.63 (0.52) 0.90 (0.83) 0.95 (0.93) 0.93 (0.90) 0.89 (0.85) Reading/8 0.77 (0.69) 0.76 (0.67) 0.74 (0.67) 0.70 (0.62) 0.90 (0.80) 0.95 (0.94) 0.91 (0.87) 0.91 (0.88) Reading/9 0.78 (0.71) 0.74 (0.66) 0.71 (0.62) 0.66 (0.57) 0.92 (0.84) 0.96 (0.94) 0.92 (0.90) 0.90 (0.86) Reading/10 0.78 (0.71) 0.75 (0.64) 0.74 (0.66) 0.72 (0.64) 0.91 (0.81) 0.96 (0.95) 0.92 (0.88) 0.90 (0.87) Science/5 0.79 (0.71) 0.74 (0.68) 0.73 (0.64) 0.67 (0.57) 0.91 (0.85) 0.95 (0.93) 0.93 (0.91) 0.90 (0.87) Science/8 0.80 (0.73) 0.71 (0.64) 0.70 (0.61) 0.65 (0.55) 0.93 (0.87) 0.95 (0.93) 0.94 (0.92) 0.91 (0.88) Science/11 0.79 (0.72) 0.73 (0.62) 0.70 (0.62) 0.68 (0.61) 0.93 (0.85) 0.97 (0.95) 0.92 (0.90) 0.90 (0.86) Writing/4 0.80 (0.73) 0.76 (0.67) 0.80 (0.74) 0.77 (0.69) 0.90 (0.79) 0.96 (0.94) 0.92 (0.89) 0.93 (0.90) Writing/8 0.79 (0.72) 0.76 (0.68) 0.77 (0.70) 0.72 (0.64) 0.91 (0.83) 0.95 (0.94) 0.92 (0.89) 0.92 (0.89) Writing/10 0.79 (0.72) 0.75 (0.64) 0.79 (0.73) 0.73 (0.66) 0.90 (0.79) 0.97 (0.95) 0.91 (0.88) 0.92 (0.89) Abbreviations: (Level of Complexity) P- = Participatory; S- = Supported; I- = Independent (Performance Levels) -B=Basic, -P = Proficient.


SECTION IV: THE VALIDITY EVALUATION Chapter 10. REVISITING THE VALIDITY EVALUATION QUESTIONS

Each of the sections in this technical report contributes important information to an argument

for validity by addressing one or more of the following aspects of the Florida Alternate Assessment:

test development, test alignment, test administration, scoring, item analyses, reliability, performance

levels, and reporting.

To measure test content validity is to determine how well the assessment tasks represent the

curriculum and standards for each content area and grade level. This is informed by the assessment

development process, including how the Levels of Complexity, test blueprints, and student evidence

align to the curriculum and standards. Viewed through this lens, as provided in the Standards for

Educational and Psychological Testing (AERA, APA, NCME, 1999), evidence based on test content

was extensively described in the development and administrations sections of the report. Content

appropriateness review processes; adherence to the test blueprint; use of standardized administration

procedures; and appropriate test administration training are all components of validity evidence

based on test content. The state provided a vehicle for extensive administrator training and a detailed

administrator manual.

Information provided in another section of this report described the scoring procedures for

the Florida Alternate Assessment as well as the processing of scoring documents.

Evidence based on internal structure was presented in the section of this report titled under

the Technical Characteristics of the Florida Alternate Assessment. Technical characteristics of the

assessments were presented in terms of item statistics, reliability measures, and decision accuracy

and consistency indices.

Methods for gathering evidence on the consequences of testing will be discussed with the

FLDOE at a later date. The FLDOE is interested in performing studies that support the external

Chapter 10—Validity Evaluation 85 2007–08 Florida Alt Technical Report

validity of the assessment through the consistency and accuracy of student performance during daily

instruction as compared to performance on the assessment, inter-rater consistency of scoring of the

assessment, and through support of the levels of complexity and the hierarchy of skills they

represent. These studies when completed will be outlined in a future technical report. For the present

time, the report shells for the Florida Alternate Assessment speak to the efforts undertaken to

promote accurate and clear information provided to the public regarding test scores. Performance

level descriptors provide users with reference points for mastery at each grade level, which is

another useful and simple way to interpret scores. The continued development of the Florida

Alternate Assessment interpretation information for parents and teachers adds to the clarity of

information provided to the public.

The evidence presented in this report supports inferences of student achievement on the

content represented in the Sunshine State Standards and Sunshine State Standards Access Points for

reading, mathematics, writing, and science for the purposes of program and instructional

improvement and as a component of school accountability. As reflected in the most recent

Standards, validity has grown to be understood as a unitary concept with content, criterion-related,

and construct validity describing three aspects of validity rather than three separate types of validity.

In addition to validity being viewed from a unitary perspective, the concept of validity has been

broadened to address issues related to social consequences and value implications of test

interpretations and uses (Messick, 1989a, 1989b). It is in the same spirit that the validity evidence in

this report is presented.

Chapter 10—Validity Evaluation 86 2007–08 Florida Alt Technical Report

REFERENCES

References 87 2007–08 Florida Alt Technical Report

REFERENCES Allen, Mary J. & Yen, Wendy M. (1979). Introduction to Measurement Theory. Belmont, CA: Wadsworth,

Inc.

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Baker, F. B. (1992). Item response theory: parameter estimation techniques. New York, NY: Marcel Dekker, Inc.

Baker, F.B., Kim, S-H.(2004). Item Response Theory: parameter estimation techniques. New York: Marcel Dekker, Inc. 2nd Edition.

Brown, F. G. (1983). Principles of educational and psychological testing (3rd Edition). Fort Worth: Holt, Rinehart and Winston.

Chicago Manual of Style (15th ed.). (2003). Chicago: University of Chicago Press.

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.

Dorans, N. J., & E. Kulick (1986). Demonstrating the utility of the standardization approach to assessing unexpected differential item performance on the Scholastic Aptitude Test. Journal of Educational Measurement, 23, 355-368.

Dorans, N. J., & Holland, P. W. (1993). DIF detection and description. In P. W. Holland & H. Wainer (Eds.) Differential item functioning (pp. 35-66). Hillsdale, NJ: Lawrence Erlbaum Associates.

Feldt, L.S., & R. L. Brennan (1989). Reliability. In R.L. Linn (Ed.), Educational measurement (3rd ed., pp. 105-146). New York: Macmillan Publishing Company.

Hambleton, R.K., & H. Swaminathan (1985). Item response theory: Principles and applications. Boston, MA: Kluwer Academic Publishers.

Hambleton, R.K., & W. J. van der Linden (1997). Handbook of modern item response theory. New York, NY: Springer-Verlag.

Holland, P. & D. T. Thayer (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. Braun (Eds.), Test validity (pp. 129-145). Hillsdale, NJ: Lawrence Erlbaum Associates.

Holland, P.W., & Wainer, H. (1993). Differential item functioning. Hillsdale, NJ: Erlbaum.

Joint Committee on Testing Practices (1988). Code of fair testing practices in education. Washington, D.C.: National Council on Measurement in Education.

Joint Committee on Testing Practices. 2004. Code of fair testing practices in education.

Kingston, N.K., Kahl, S.K., Sweeney, K.P., and Bay, L. (2001). Setting Performance Standards Using the Body of Work Method. In Cizek, G.J. (ed.). Setting Performance Standards: Concepts, Methods, and Perspectives. Mahwah, NJ: Lawrence Erlbaum.


Kolen, M.J., & Brennan, R.L. (1995). Test Equating: Methods and Practices. New York: Springer-Verlag.

Livingston, S. A., & Lewis, C. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32, 179-197.

Lord, F.M. & Novick, M.R. (1968). Statistical Theories of Mental Test Scores. Reading, MA: Addison-Wesley.

Muraki, E. & R. D. Bock (2003). PARSCALE 4.1. Lincolnwood, IL: Scientific Software International.Subkoviak, M.J. (1976). Estimating reliability from a single administration of a mastery test. Journal of Educational Measurement, 13, 265-276.

Petersen, N.S., Kolen, M.J., & Hoover, H.D. (1989) Scaling, norming, and equating. In R.L. Linn (Ed.), Educational measurement (3rd ed., pp. 221-262).

Stocking, M. L., & Lord, F. M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7, 201-210

Stout, W. F. (1987). A nonparametric approach for assessing latent trait dimensionality.Psychometrika, 52, 589-617.

Stout, W. F., Froelich, A. G., & Gao, F. (2001). Using resampling methods to produce an improved DIMTEST procedure. In A. Boomsma, M. A. J. van Duign, & T. A. B. Snijders(Eds.), Essays on Item Response Theory, (pp. 357-375). New York: Springer-Verlag.

Washington, DC: American Psychological Association. Available for download at http://www.apa.org/science/fairtestcode.html.

Zhang, J., & Stout, W. F. (1999). The theoretical DETECT index of dimensionality and its application to approximate simple structure. Psychometrika, 64, 213-249.


APPENDICES

Appendices 91 2007–08 Florida Alt Technical Report

APPENDIX A—FLORIDA STAKEHOLDER LISTS

Appendix A—FL Stakeholders List 93 2007–08 Florida Alt Technical Report


Reading Extensions Writing Team, July 2005 Rita Ammons

ESE Middle School Teacher, Lee County Schools Elizabeth Barton

Escambia County Schools Martha Berman

ESE Elementary Teacher, Miami-Dade County Holly Brozi

Vocational Education, Monroe County Ginger Copeland

ESE Middle School Teacher, Levy County Blanquite Cruz

ESOL Middle School Specialist, Lake County Meredith Duraslanti

ESE Middle School Teacher, Hardee County Patricia Fisher

Lake County Schools RhaoNan Florio

ESE Middle School Teacher, Marion County Bonnie Hammer

ESE Coordinator, Santa Rosa County Schools Trace Hines

District Administrator, Volusia County Victoria Hurley

Middle School Reading Teacher, Okaloosa County Kristen Landry

ESE PreK Teacher ,Wakulla County Randy LaRusso

Alternate Assessment Coordinator, Brevard County Maria Mesa

ESE High School Teacher, Miami-Dade County Mike Muldoon

Florida Inclusion Network Facilitator, USF-St. Petersburg Jodi O‘Meara

Curriculum Specialist, Manatee County Mike Perduto

Vocational Education, Brevard County Constance Pridgeon

District Curriculum Coordinator, Gilchrist County Dee Ragar

High School Reading Teacher, Sumter County Jamie Ruccolo


Middle School Reading Teacher, Broward County Susan Seal

Middle School Reading Teacher, St. Lucie County Claire Smith

ESE Coordinator, Marion County Betty Spires

ESE Resource Specialist, Walton County LaJune Stephens

ESE Elementary Teacher, Taylor County Marie Thienel

ESE Middle School Teacher, Osceola County C.J. Weyrich œ

ESE Elementary Teacher, Franklin County

Writing Extensions Writing Team, 2005 Mary Asciutto

Elementary Teacher, Highlands County Anita Askew

Reading Teacher, Gulf County Schools Cindy Bania œ Pinellas County Schools

Alternate Assessment Coordinator and District Administrator Carolyn Baxter

Jackson County Schools Betsy Botts

Santa Rosa County Schools Sclena Brantley

Middle School Mathematics Teacher, Pasco County Daniel Brownyard

Elementary Teacher, Polk County Mevldine Carter

Escambia County Schools Cathy Chelberg

ESE Elementary Teacher, Seminole County = Badonna Dardis

Assistant Principal, Polk County Phyllis DeSesso

ESE Middle School Teacher, Hernando County = Marjorie Garland

Middle School Language Arts Teacher, Martin County = Margaret Hanka

ESE Middle School Teacher, Pasco County = Libby Harris

ESE Middle School Teacher, Columbia County =


Bonnie Hernandez ESE Elementary Teacher, Bay County

Melissa Homan ESE High School Teacher, St. Lucie County

Sheila Newell ESE and Elementary Teacher, Polk County

Ellen Quartano High School English Teacher, Orange County

Ellen Sage Manatee County Schools

Nancy Shackelford Alternate Assessment Coordinator, Hendry County

Holly Ward High School English Teacher, Volusia County

Diane Wilen ESOL Elementary Specialist, Broward County

Reading and Language Arts Access Points Consultants Stacie Whinnery

University of West Florida Laurie Brennan

Elementary ESE Teacher, Leon County Tanja Estes

Secondary ESE Teacher, Leon County Michele Gould

Middle School ESE Teacher, Leon County Elaine Harrison

High School ESE Teacher, Leon County


Alternate Assessment Advisory CommitteeMembers

Name Representing Denise Rusnak Broward County Rosalind Hall Levy County Debi Dukes Union County Janet Hurley Manatee County Bruce Harrison Leon County Jo Wilson Gilchrist County Jill Brookner Miami-Dade County Bill Elkin Lee County Barbara Hardy Bay County Ruth Casias Clay County Natalie Roca Sarasota County; Malcolm Thomas Escambia County Conney Dahn Martin County Patti Crooks Martin County Patty Rusler Orange County Sam Thompson Leon County Marjorie Murray Seminole County Stacie Whinnery University of West Florida Susan Tucker Duval County Brian McMahon Indian River County Jessica Webb Madison County Cornelia Orr Florida Department of Education Bambi Lockman Florida Department of Education Sheryl Sandvoss Florida Department of Education Kris Ellington Florida Department of Education Victoria Ash Florida Department of Education Carol Allman Consultant


Other Attendees Name Representing

Anne Chartrand Facilitator, Southeast Regional Resource Center Karen Denbroeder Florida Department of Education Angela Nathaniel Florida Department of Education Chris Sarno Piedra Data Services Adalis Sanchez Piedra Data Services Gail Best Accountability and Assessment for Students with

Disabilities Bennett Buckles Accountability and Assessment for Students with

Disabilities Donna Hubbs Accountability and Assessment for Students with

Disabilities Leigh Palka Accountability and Assessment for Students with

Disabilities Susan Izard Measured Progress Rebecca Walk Measured Progress Brook Loch Measured Progress Jessica Demmons Measured Progress

Bias Review Committee

Name Role Gender Ethnicity District/Size Area of State

*Evelyn Langston ESE Dir. F 2 Putnam/MS Northeast

*Dawn Saunders FLDOE F 2 Leon/M North

*Peggy Harter Assistive Tech Specialist

F 1 Leon/FDLRS North

*Frank Wadler ESE Teacher; HI, VI, ESOL, Reading, MH

M 1 FSDB Northeast

*Lora McCalister-Cruel

ESE Teacher F 2 Bay/M Northwest

*Tony Silva School Psychologist M 3 Palm Beach/VL Southeast

*Gina Horton Instructional Support

Teacher/Specialize d Curriculum

F 1 Orange/VL Central

*Richard Stokes ESE Teacher, HS M 2 Pasco/L West Central

*Deland Innocent District Bilingual Assessor

M 6 Miami-Dade/VL South


CONTENT REVIEW Reading Team Name/Role Gender District/Size Area of State

Elementary Constance Pridgeon; Gen ed F Gilchrist/S North Central

Sheila Newell; gen. ed 3rd F Polk/L Central

Tina Mayes; ESE K-5 (mostly autistic students) F Osceola/L Central

Joan Radford; Reading coach/former ESE F Hamilton/S North Central

Middle Anita Askew; Gen. ed., MS F Gulf/S Northwest

Sarah Trauger; Gen. ed Reading/ ESE background F Duval/VL Northeast

Cheryl Dudash; general ed; reading coach F Broward/VL Southeast

Jeanette Gresham; ESE MS F Pasco/L West Central

Rosalind Hall; ESE director F Levy/S Central

Debra Doster; ESE F Volusia/L Central East

High School Christine Richards; District HS Reading Specialist F St. Lucie/M Central East

Elisa Giordano; Gen Ed F Broward/VL South East

Susan Seal; Gen. ed, MS/HS F St. Lucie/M Central East

Llidia Velado; ESE F Volusia/L Central East

Richard Stokes; ESE M Pasco/L West Central


Math Team Name/Role Gender District/ Size Area of State

Elementary Kim Dimaggio; Gen Ed F Orange/VL Central

Tiffany Stanley; Gen Ed F Orange/VL Central

Terrilynn Latour, Gen Ed F Broward/VL South East

Stephanie Skwyra; ESE 3-5 F Osceola/L Central

Leslie Rogers; ESE F Pinellas/VL Central West

Middle Angel Johnson; ESE MS/HS F Pasco/L West Central

Sharon Whitaker; ESE F Okeechobee/MS Central East

Delia H Pogorzelsa; Gen Ed F Leon/M North Central

Violette Espinoza; Gen Ed F Broward/V South East

Mary Lou Darby; Assessment F Santa Rosa/M North West

Sheribeth Marquis; Gen Ed; ESE F Leon/M North Central

High School Carla Frazier; Data Analysis F Manatee/L Central West

Jim Devune; Gen Ed M Flagler/MS North East

Merrill Levine; ESE F Broward/VL South East

Bridget White; ESE F Pasco/L West Central

Amy Modesto; ESE/HS F Orange/VL Central


Writing Team Name/Role Gender District/ Size Area of State

Elementary Bonnie Hernandez; ESE K-2 F Bay/M Northwest

Diane Jackson; ESE Elem. F Pasco/L West Central

Allison Owens; ESE F Leon/M North Central

Lisa Woulard-Akinsola; Gen Ed F Leon/M North Central

Myra Harp; Gen Ed, Ele F Leon/M North Central

Rose Mitchell-Freeman F

Middle Rebecca Jones; Gen Ed F Orange/VL Central

Kelly Milrot; Gen ed MS/LA F Okeechobee/MS Central East

Lee Ann Goble; ESE F Santa Rosa/M North West

Cathy Chellberg; ESE K-5, MS F Seminole/L Central

Sarah Rapparport; ESE F Broward/VL Southeast

High School Ellen Quartano; ESE HS LA F Orange/VL Central

Laura Harbin; ESE 9-10 F Osceola/L Central

Holly Bach; Gen Ed F Hendry/MS South West Tina Kennon-McIntyre, Director, ESE F Suwannee/S North Central

Donna Buckner; gen ed LA FCAT committee F Polk/L Central


Science Team Name/Role Gender District/Size Area of State

Elementary Martha Berman; ESE Curr. F Dade/VL South

Patricia Rusler; ESE K-5 F Orange/VL Central

Celeste Cobelens; ESE F Pinellas/VL Central West

Mary Flynn; Gen Ed F Broward/VL South West

Sharon Sams; Gen Ed F Leon/M North Central

Middle Monica Griffey; MS/Sc& Ma F FSDB Northeast

Susan Johnson; ESE F Wakulla/S North Central

Mary Jurczak; ESE F Volusia/L Central East

Christian L’eon; ESE F Pinellas/VL Central West

Shannan Romer; Gen Ed F Calhoun/ S North West

Tavia Marez; ESE F Okaloosa/M North West

High Carmeline Williams; Gen Ed F Collier/ L South West

Jenn Tibbitts; ESE F Leon/M North

Debbie Morrow; Center School Teacher F Charlotte/MS South West

Malcolm Thomas; Assessment Director M Escambia/L North Central

Aaron Bach; Gen Ed M Hendry/MS South West

Fill in Bennett Buckles M St. Lucie/M Central East


APPENDIX B—SAMPLE ITEM: FIELD TEST FORMAT

Appendix B—Sample Item Field Test Format 105 2007–08 Florida Alt Technical Report






APPENDIX C—SURVEYS AND RESULTS

Appendix C—Survey Results 113 2007–08 Florida Alt Technical Report

CONTENT REVIEW COMMITTEE FEEDBACK

Reading Strongly Disagree Disagree

Neither Agree Nor Disagree Agree

Strongly Agree

1 2 3 4 5 Overall the review worked well 0% 0% 0% 36% 64% The overview and item walk through were helpful 0% 7% 0% 21.5% 71.5%

The process for feedback and recommendations worked well. 0% 0% 7% 14% 79% The location of the meeting and facilities worked well 0% 0% 0% 7% 93%

Science Strongly Disagree Disagree


Strongly Agree

1 2 3 4 5 Overall the review worked well 0% 0% 0% 25% 62.5% The overview and item walk through were helpful 6% 0% 0% 25% 69%


Math Strongly Disagree Disagree


Strongly Agree

1 2 3 4 5 Overall the review worked well 0% 0% 0% 47% 53% The overview and item walk through were helpful 0% 0% 0% 33% 67%


Writing Strongly Disagree Disagree


Strongly Agree

1 2 3 4 5 Overall the review worked well 0% 0% 0% 71% 29% The overview and item walk through were helpful 0% 0% 0% 57% 43%



TRAIN-THE-TRAINER FEEDBACK

September 5th No Response

Strongly Disagree Disagree

Neither Agree

nor Disagree Agree

Strongly Agree

Overall the training worked well. 0% 0% 0% 6.5% 37% 56.5%

The questions activity was helpful. 0% 0% 4% 11% 41.5% 41.5%

The Scavenger Hunt Activity was helpful. 2% 2% 2% 20% 37% 39% The overview and manual walk through were helpful. 2% 0% 2% 2% 26% 68%

The Open Response Activity was helpful. 0% 0% 2% 15% 39% 44%

Seeing the administration demonstration was helpful. 7% 0% 2% 15% 26% 50%

The sample Items were helpful. 2% 0% 0% 6.5% 32.5% 5%

The questions I had about the new Florida Alternate Assessment were answered. 0% 0% 11% 13% 43% 33%

The materials provided were helpful. 0% 0% 2% 2% 33% 63%


September 6th No Response 1 2 3 4 5

Overall the training worked well. 2.5% 0% 2.5% 0% 51% 44%

The questions activity was helpful. 0% 0% 0% 5% 51% 44.0%

The Scavenger Hunt Activity was helpful. 0% 0% 0% 12% 49% 39% The overview and manual walk through were helpful. 0% 0% 0% 2.5% 36.5% 61%

The Open Response Activity was helpful. 2% 0% 5% 5% 61% 27%

Seeing the administration demonstration was helpful. 12% 0% 3% 5% 34% 46%

The sample Items were helpful. 0% 0% 0% 2.5% 41.5% 56%

The questions I had about the new Florida Alternate Assessment were answered. 2.5% 2.5% 5% 5% 58% 27%

The materials provided were helpful. 0% 0% 0% 0% 49% 51%


FIELD TEST ON-LINE SURVEY RESULTS

Total number of full/completed years teaching:

Total number of full/completed years teaching students with significant cognitive disabilities:


I am instructing my students in the Access Points in English Language Arts.

I am instructing my students in the Access Points in Mathematics.

I attended a training.

The training was:

This was enough time for me to learn how to administer the assessment.


The training prepared me for administering the assessment.

The administration directions in the manual were easy to follow.

The samples in the manual adequately gave me a feel for what to expect from the administration.

The Rubric/Content Specific Direction Sheet was helpful during administration.


Once I became familiar with the assessment it became easier to administer.

Overall, it was clear how to administer the assessment items.

Overall, the graphics for the assessment items were appropriate.

The auxiliary and teacher-gathered materials were appropriate to the items.

The auxiliary materials were provided to me already cut out.


I was able to easily organize the auxiliary materials for administration.

How many hours did it take to administer the reading assessment?

How many days did you use to administer the reading assessment?

How many hours did it take to prepare and organize all of the materials (auxiliary and teacher-gathered) for reading?


The item prompt —show me/tell me“ was easily replaced to match the student‘s response mode.

Overall, the process used in administration of the items was new to the student.

Overall, the content assessed in the items was new to the student.


OPERATIONAL ON-LINE SURVEY RESULTS

Total number of full/completed years teaching:

Total number of full/completed years teaching students with significant cognitive disabilities:

I have been instructing my students in the Access Points in English Language Arts.


I am instructing my students in the Access Points in Mathematics.

I participated in the Fall 2007 field test of the Florida Alternate Assessment.

I attended additional training since the training for the field test in October.

This was enough time for me to learn about the changes in the administration of the assessment.

The training prepared me for administering the assessment using the new format.


The revised administration directions in the manual were clearer and easier to follow.

The revised samples in the manual adequately gave me a sense of what to expect from the administration.

Having the Quick Reference Sheet a different color helped to remind me to use it during administration of the assessment.

The Quick Reference Sheet was beneficial in the administration of the assessment.


Once I became familiar with the assessment it became easier to administer.

Overall, it was clear how to administer the assessment items.

Overall, the graphics for the assessment items were appropriate.

The auxiliary and teacher-gathered materials were more manageable for the Operational Test.


It was easier to organize the auxiliary materials for the Operational administration.

How many hours did it take to administer the reading assessment?

How many days did you use to administer the reading assessment?


APPENDIX D—SAMPLE ITEM: OPERATIONAL TEST FORMAT

Appendix D—Sample Item Test Format 129 2007–08 Florida Alt Technical Report



APPENDIX E—REPORT SHELLS

Appendix E—Report Shells 133 2007–08 Florida Alt Technical Report

State Report Page 1 Only (FRONT)


State Report (BACK)


Report Sample: District Report (FRONT)




Report Sample: District Report (BACK)


Report Sample: School Report (FRONT)



Report Sample: School Report (BACK)


Report Sample: Individual Student Report (FRONT)


Report Sample: Individual Student Report (BACK)


APPENDIX F—PARENT AND TEACHER BROCHURES

Appendix F—Parent & Teacher Brochure 147 2007–08 Florida Alt Technical Report

Dr. Eric J. Smith Commissioner of Education

Understanding the


Information for Parents 2007 – 2008


How will the assessment results be used? Your child‘s results can be used to: • assist the Individual Educational Plan (IEP) team in developing annual goals and objectives; • inform instructional planning; • improve student achievement in the classroom; • monitor progress from year to year; • determine teacher training needs; and • ensure that students are taught the knowledge and skills outlined in the Sunshine State Standards

Access Points.

What can I do as a parent to prepare my child for the Florida Alternate Assessment? • Make sure that your child is well rested. • Provide your child with a well-rounded diet. This will help ensure a healthy body and a healthy,

active mind. • Encourage your child to do their best.

How can I get more information about the Florida Alternate Assessment? More information about the Florida Alternate Assessment can be obtained by talking to your child‘s teacher or school district‘s ESE Administrator. You can also contact the Florida Department of Education Bureau of Exceptional Education and Student Services at 850.245.0475.

Translations of this publication are available in Spanish and Haitian Creole through your district‘s ESE Administrator, or you can download copies of this brochure at http://www.fldoe.org/ese/.


The Florida Alternate Assessment The Florida Alternate Assessment is a performance-based assessment, not a paper and pencil test. It is designed for students with significant cognitive disabilities for whom participation in the Florida Comprehensive Achievement Test (FCAT) is inappropriate, even with accommodations.

Who determines if my child participates in the Florida Alternate Assessment? • Decisions regarding the type of assessment to be administered are based upon the student‘s

individual needs and determined by the Individual Educational Plan (IEP) team which includes you as the parent, teachers, administrators, and service providers.

What are the academic expectations of students with disabilities? • Florida has a standards driven system for all students. • The Sunshine State Standards and Access Points for Students with Significant Cognitive

Disabilities drive the curriculum, instructional strategies and assessment. • Student success in achieving the standards is assessed in one of three ways:

FCAT without accommodations FCAT with accommodations Florida Alternate Assessment for Students with Significant Cognitive Disabilities

Sunshine State Standards

and Access Points

• FCAT without accommodations

• FCAT with accommodations

• Florida Alternate Assessment

Curriculum and Instructional

Strategies

What are Access Points? • Access Points reflect the key concepts of the Sunshine State Standards with reduced levels of

complexity. They ensure access to the essence or core intent of the standards that apply to all students in the same grade.

For more information about the Access Points, visit www.flstandards.org.

What are the levels of complexity? • Access Points describe the knowledge and skills required at each grade level. • Each Access Point has three levels of complexity. • The three levels of complexity are: Independent (In), Supported (Su), and Participatory (Pa), with

Participatory level being less complex.

In Su Pa

More Complex Less Complex


Dr. Eric J. Smith Commissioner of Education

Facts About the Florida Alternate Assessment

Information for Teachers

2007 – 2008


Who can administer the Florida Alternate Assessment? • The Florida Alternate Assessment is administered to each student individually by the student‘s

special education teacher. If this is not possible, the test administrator must be a certified teacher or other licensed professional who has worked extensively with the student, and is trained in the assessment procedures.

When is the Florida Alternate Assessment Administered? • The Florida Alternate Assessment is administered annually and assesses students in Reading

(grades 3 œ 10), Mathematics (grades 3 œ 10), Writing (grades 4, 8, and 10), and Science (grades 5, 8, and 11).

How can teachers use the assessment results? Results of the Florida Alternate Assessment show educators how students with significant cognitive disabilities are progressing toward learning the knowledge and skills contained in the Sunshine State Standards Access Points.

The results can be used to assist the IEP team in developing annual goals and objectives. The IEP team should examine the results in conjunction with other informationœsuch as progress reports, report cards, and parent and teacher observationsœto see what additional instruction is needed and in what areas.

The results can also be used to improve instructional planning. For example, a student exhibiting skills within the participatory level of complexity at an advanced performance level may be ready for more difficult work and will most likely be instructed on a combination of Access Points at both the participatory and supported levels. Students‘ scores may also indicate a need to have the curriculum adjusted or for students to be provided with additional supports and learning opportunities.

Overall the assessment scores can be used to plan instruction in order to improve student outcomes.

Do the Florida Alternate Assessment results count toward my school making Adequate Yearly Progress (AYP)? Yes, a student‘s alternate assessment score is included in the school and district‘s AYP calculation.

For more information about the Florida Alternate Assessment, contact your school district’s Exceptional Student Education (ESE) Administrator or the Florida Department of Education, Bureau of Exceptional Education and Student Services, at 850.245.0475.


What is the Florida Alternate Assessment? The Florida Alternate Assessment is a performance-based assessment. It is designed for students with significant cognitive disabilities for whom participation in the Florida Comprehensive Assessment Test® (FCAT) is not appropriate, even with accommodations. Using the Alternate Assessment Participation Checklist, Individual Educational Plan (IEP) teams may determine that the student should be assessed on the Florida Alternate Assessment if the following criteria are met: • The student‘s demonstrated cognitive ability prevents him/her from completing required coursework and

achieving the Sunshine State Standards even with appropriate and allowable course modifications, and • The student requires extensive direct instruction to accomplish the application and transfer skills and

competencies needed for domestic, community living, leisure, and vocational activities (Rule 6A-1.0943 (1)(a) 1-2, Florida Administrative Code [FAC.]).

The Florida Alternate Assessment Participation Checklist can be obtained at http://info.fldoe.org/docushare/dsweb/Get/Document-1858/AA%20Check.pdf.

What are Access Points?

The Florida Alternate Assessment is based on the Sunshine State Standards Access Points for students with significant cognitive disabilities. The Access Points drive the curriculum, instructional strategies, and, ultimately, student outcomes as measured by the assessment.

Sunshine State Standards

and Access Points


Curriculum and Instructional

Strategies

Access Points are aligned to the Sunshine State Standards with reduced levels of complexity. They ensure access to the general curriculum at each grade level. • Access Points describe the knowledge and skills required at each grade level. • Access Points are written within three levels of complexity: Independent (In), Supported (Su), and

Participatory (Pa), with Participatory level being least complex.

In Su Pa

More Complex Less Complex


For more information about the Access Points, visit Florida’s Department of Education Web site at www.flstandards.org.

What assessment results are provided to teachers and parents? Individual student reports contain the following information:

• The grade level and content areas tested. • The student‘s total score for each content area determines the level of complexity and performance level.

The three levels of complexity are participatory, supported, and independent; and within these three levels of complexity are three performance levels: basic, proficient, and advanced.

To assist parents in understanding the Florida Alternate Assessment scoring system, please refer to the Administration and Scoring Process Flow Chart and the Scoring Rubric and Directions section in your Florida Alternate Assessment Test Administration Manual.

Participatory Level of

Complexity

Basic Level of

Performance

Proficient Level of

Performance

Advanced Level of

Performance

Supported Level of

Complexity

Basic Level of

Performance

Proficient Level of

Performance

Advanced Level of

Performance

Independent Level of

Complexity

Basic Level of

Performance

Proficient Level of

Performance

Advanced Level of

Performance


APPENDIX G—ITEM SPECIFICATIONS DOCUMENT

Appendix G—Item Specifications Document 155 2007–08 Florida Alt Technical Report


Florida Alternate Assessment Test Designs, Blueprints and Item Specifications for

Reading, Writing, Mathematics and Science

May 2007

Prepared by Measured Progress for the Florida Department of Education


Table of Contents

Overview ………………………………………………………………………….. 1

Items ………………………………………………………………………………. 2

Test Booklet Components Item Components Depth of Knowledge

Number of Items by Content and Grade Level

Reading …………………………………………………………………………… 6

Design Blueprint Passage Specifications

Writing …………………………………………………………………………….. 13

Design Blueprint

Mathematics ……………………………………………………………………... 16

Design Blueprint

Science …………………………………………………………………………… 20

Design Blueprint

Item Specifications ……………………………………………………………. 23


Overview

The new alternate assessment design for Florida is based on the revised Sunshine State Standards with three levels of access points (independent, supported, and participatory), to provide a tiered entry to the assessment for students at the various levels. This is critical as educators seek to provide access to the general education curriculum and foster higher expectations for the wide diversity of students with significant cognitive disabilities.

These assessments will contain primarily performance tasks made up of selected response options and some open response options. We therefore propose to use the access points to develop an assessment blueprint that will serve as the foundation for structured student performance tasks. Our proposed design is an innovative approach that provides test administrators with structured tasks comprised of item sets that reflect typical classroom activities and embed items frequently containing three response options for students to select with the communication systems they normally use.


Items Within each item set, we propose to address the student‘s presentation and response mode. Items move from pre-symbolic communication presentation and response to concrete symbolic and finally abstract symbolic. The levels of communication increase with the access points, but may also vary within the access point. Some access points require the student to be at an abstract symbolic level of communication; most likely this occurs at the independent level. All items are initially set up as multiple choice and move toward an open response at the highest level.

It is our experience that students who use communication supports are assessed more accurately when they are provided with structured response options within a performance task. Students who have greater access to verbal or written responses will be able to respond to open or constructed response items. For example, when a non-verbal student with mobility challenges is asked a question and presented with the choices for the answer, that student may use eye gaze to indicate the preferred choice, hit a switch from among several pre-programmed switches, point to one choice, etc.

Items that require a constructed response or multi-step performance, such as organizing pictures to show the order of events in a story, are often more challenging for this population of students. Therefore, we propose to incorporate an element of universal design in the development of alternate performance tasks to build a test on which all students, even those with the most significant communication challenges, have the opportunity to respond accurately: we recommend presenting three options to students when multiple response options are required. This limits the cognitive load of the item and adheres to recommendations of Haladyna and Downing1, who contend that more than three acceptably performing distracters are rarely found.

Within each item set, each of the three access points will be addressed. Each student starts at the participatory level. A student completing the participatory level item accurately without assistance moves on to the supported level item. In this way, the student moves up through the access points as long as he or she is able to respond accurately and independently. At the participatory level item only, for a student who is unable to complete the participatory level item accurately and independently scaffolding will occur. The student will be presented the item again with one distracter removed, if the student is able to accurately respond he/she will be scored at two points. If the student is still unable to accurately respond the item is presented again with another distracter removed (leaving only the correct answer) and the student is asked to actively engage with the correct answer. At any point within the Participatory Level item, if the student will not engage or actively refuses the student will score a zero point.

The student receives the final score for the item set based on the level at which it was answered correctly. For example, if the student is unable to complete the item at the supported level, he or she retains the three-point score from the participatory level. However, if he or she is able to complete the supported item, the teacher will next administer the independent level item. If the student is unable to complete the independent

1 Educational and Psychological Measurement, Vol. 53, No. 4, 999-1010 (1993) DOI: 10.1177/0013164493053004013 © 1993 SAGE Publications


item accurately, a score of six points is awarded. However, if the student completes the independent item accurately, the teacher will record a score of nine points.

0 1 2 3 6 9

No response, student actively refuses or does not engage at

any point during the Participatory

Level

Student responds

correctly after the removal of two distracters

at the Participatory

Level

Student responds

correctly after the removal of one distracter

at the Participatory

Level

Student responds

correctly at Participatory

Level

Student responds

correctly at Supported

Level

Student responds

correctly at Independent

Level

We will provide educators with auxiliary materials, such as sentence strips, when they are specifically needed to complete an item. The test booklets will include scripting for the educators to follow as they administer the assessment, increasing procedural reliability. We also encourage the use of classroom materials that students are familiar with, giving students the best opportunity to demonstrate their knowledge and skills.

Test Booklet Components Each content area section of the test booklet will begin with an overview of the strands and standards being assessed at that grade and a list of classroom materials that the educator should gather to augment the materials sent with the test booklet (i.e. for mathematics, counting blocks may be required.)

The test booklet itself will include item sets that describe the materials provided, materials needed from the classroom, teacher scripting at each access point, the expected student response, the access point being assessed, and a place to score the student on each item set. The test booklet can further provide recommended stopping points for students who may easily fatigue during testing.

We will design the test booklets with educators in mind, understanding that teachers need to easily refer to the test booklets during administration and scoring.

Item Components Each item set will include an overview, the access points to be assessed, and the materials needed. The components for each item set are:

Materials Teacher Will Student Will Scoring Access Point

• The Materials column outlines for the educator which materials will be needed for the item. Materials that are provided for the teacher and materials the educator may need to gather from the classroom are identified. Further explanation for supports and adaptations a student may need can also be clarified in this column to assist the teacher during administration. The materials provided in this column start at a pre-symbolic level and progress to an abstract symbolic level. Graphics will be named for teachers to use standardized terminology as needed.

• The Teacher Will column consists of a clear set of directions for setting up the item and scripting for what the teacher should ask the student.


• The Student Will column indicates the response that the educator needs to look for from the student, taking into consideration the mode appropriate for each student.

• The Scoring column provides a space for the educator to mark the score the student received on the item.

• The Access Point column lists the access point that the item is targeting.

Depth of Knowledge We propose to include specifications on the depth of knowledge (DOK) assessed by each item at each access level. We can do this as part of the teacher review process. We propose to use the extended Bloom‘s Taxonomy outlined by Browder in —Links for Academic Learning: An Alignment Protocol for Alternate Assessments Based on Alternate Achievement Standards.“

Level Depth of Knowledge Descriptor 1 Attention ( touch, look, vocalize, respond, attend)

2 Memorize/recall (list, describe (facts), identify, state, define, label, recognize, record, match, recall, relate)

3 Performance (perform, demonstrate, follow, count, locate, read)

4 Comprehension (explain, conclude, group/categorize, restate, review, translate, describe (concepts), paraphrase, infer, summarize, illustrate)

5 Application (compute, organize, collect, apply, classify, construct, solve, use, order, develop, generate, interact with text, implement)

6 Analysis, Synthesis, Evaluation (pattern, analyze, compare, contrast, compose, predict, extend, plan, judge, evaluate, interpret, cause/effect, investigate, examine, distinguish, differentiate, generate)

Number of Items by Content and Grade Level Each content/grade level operational test will be made up of 24 items. The test design and blueprint will vary by content area and will be described in the content area sections that follow.

Grade Reading Mathematics Writing Science Total # Test

Booklet Items

3 24 24 48 4 24 24 24 72 5 24 24 24 72 6 24 24 48 7 24 24 48 8 24 24 24 24 96 9 24 24 48 10 24 24 24 72 11 24 24

Total Items

192 192 72 72 528


Reading

Design

The reading design consists of two strands that will be measured by the items in the test. In addition, two standards for each of the two strands will be identified for assessment. Each standard will consist of six items for a total of twenty-four reading items.

Strand 1 Strand 2 Standard 1 Standard 2 Standard 1 Standard 2

6 Items 6 Items 6 Items 6 Items 24 Items

Blueprint

In developing the test blueprint for reading, Measured Progress staff examined several documents:

• FCAT Reading 2006 Grades 3-10 Test Focus • FCAT Reading Test Item and Performance Task Specifications • FCAT Summary of Tests and Design, September 2005 • Draft FCAT Writing + Test Item Specifications, Grades 3-12 © 2005 Florida

Department of Education • Florida‘s 2006 Sunshine Standards for K-12 Reading and Language Arts. • Language Arts Draft Crosswalk, Grades 3-10

We examined the FCAT Reading 2006 Test Focus and noted the benchmarks that were covered. We mapped these benchmarks on the old standards and then used the Language Arts Draft Crosswalk to map the standards to the 2006 Sunshine Standards for K-12 Reading and Language Arts. This showed us the distribution of standard coverage against the 2006 Sunshine Standards. We also noted the Access points for the particular benchmarks in the General Education Frameworks. These notations will confirm the alignment of the access points on which we test the students with significant cognitive disabilities to the indicators on which we test general education students. The items for the Florida Alternate Assessment will be written to the Sunshine Standards, using the access points that have been submitted for State Board of Education approval.

Based on our analysis of coverage in the FCAT, the two Reading Strands that Measured Progress suggests for coverage are Reading Process and Literary Analysis. Each of these strands has multiple standards and varied grade level distribution in the FCAT.


In Reading Process, the two standards covered most across grade levels are Vocabulary and Reading Comprehension. Reading Comprehension is the purpose of reading; therefore, it is sensible to test all students on this standard. Learning vocabulary skills at the lower grades allows students to become adept in growing their reading vocabulary. At grades 9-10, however, the Crosswalk pointed to concepts not applicable in the Old Standards: Strand 3: Information and Media Literacy. Therefore, we selected this new strand that synthesizes many of the benchmark skills tested in earlier grades. For the Literary Analysis we follow the FCAT balance of fiction and nonfiction with the particular grade level emphases.

The distribution for each benchmark will be consistent with the distribution on the FCAT. Note: not every standard and benchmark is tested in the FCAT.


Strand 1 Reading Process: Grade 3

Grade 4

Grade 5

Grade 6

Grade 7

Grade 8

Grade 9

Grade 10

Standard 1: Vocabulary Development -The student uses multiple strategies to develop grade appropriate vocabulary. LA.7.1.6.1 - use new vocabulary that is introduced and taught directly …. 2 2

LA.7.1.6.3 - use context clues to determine meanings of unfamiliar words …. 2 2

LA.6.1.6.4 - categorize key vocabulary and identify salient features …. 6 LA.4.1.6.5 - relate new vocabulary to familiar words…. 1 1 LA.4.1.6.6 - identify —shades of meaning“ in related words (e.g., blaring, loud)….

1

1

2

2

LA.3.1.6.7 œ The student will use meaning of familiar base words and affixes (prefixes and suffixes) to determine meanings of unfamiliar complex words, …

2 2 1

LA.3.1.6.8 - The student will use knowledge of antonyms, synonyms, homophones, and homographs to determine meanings of words .… 2 2 1

LA.3.1.6.10 - The student will determine meanings of unfamiliar words by using a dictionary, thesaurus, and digital tools…. 2 2

Standard 2: Reading Comprehension - The student uses a variety of strategies to comprehend grade level text.

LA.3.1.7.2 - The student will identify the author‘s purpose (e.g., to inform, entertain, or explain) in text and how an author‘s perspective influences text ….

2 2 2 2 2 2 3 3

LA.3.1.7.3 - determine explicit ideas and information in grade-level text, including but not limited to main idea, relevant supporting details, strongly implied message and inference, and chronological order of events ….

2 2 2 2 2 1 1 1

LA.3.1.7.5 - The student will identify the text structure an author uses (e.g., comparison/contrast, cause/effect, and sequence of events) and explain how it impacts meaning in text….

2 2 2 2 1 2

LA.3.1.7.7 - The student will compare and contrast topics, settings, characters, and problems in two texts …. 1 1 2 2


Strand 2 Literary Analysis: Grade 3

Grade 4

Grade 5

Grade 6

Grade 7

Grade 8

Grade 9

Grade 10

Standard 1: Fiction – The student identifies, analyzes, and applies knowledge of the elements of a variety of fiction and literary texts to develop a thoughtful response to a literary selection. LA.4.2.1.2 - identify and explain the elements of plot structure, including exposition, setting, character development, problem/resolution, and theme in a variety of fiction;

3 6

6 6

LA.910.2.1.5- describe, discuss, and analyze an author‘s use of literary elements (i.e., theme, point of view, characterization, setting, plot), and explain and analyze different elements of figurative language (i.e., simile, metaphor, personification, hyperbole, symbolism, allusion, imagery) in multiple literary selections…

6 6

LA.3.2.1.6 - The student will write a book report or review that identifies the main idea, character(s), setting, sequence of events, and problem/solution;

6 3 6

Standard 2: Non-Fiction- The student identifies, analyzes, and applies knowledge of the elements of a variety of non-fiction, informational, and expository tests to demonstrate an understanding of the information presented. LA.3.2.2.2 - The student will use information from the text to answer questions related to explicitly stated main ideas or relevant details …. 3 3 3 3 3 6 6

LA.3.2.2.3 - The student will organize information to show an understanding of main ideas within a text through charting, mapping, or summarizing ….

3 3 6 3 3 3

Strand 3 Grades 9–10: Information and Media Literacy Grade 3

Grade 4

Grade 5

Grade 6

Grade 7

Grade 8

Grade 9

Grade 10

Standard: 910.6 The student uses a systematic process for the collection, processing and presentation of information. LA.910.6.2.2 - organize, synthesize analyze and evaluate the validity and reliability of information from multiple sources (including primary and secondary sources) to draw conclusions using a variety of techniques, and correctly use standardized citations;

3 3

LA.910.6.2.3 - write an informational report that integrates information and makes distinctions between the relative value and significance of specific data, facts, and ideas;

3

3

(The grade level number in the benchmark shows the grade level at which the benchmark is first tested; the benchmark may expand at higher grade levels; hence the ellipses. Numbers indicate how many item sets will be written at each grade level for each benchmark.


Passage Specifications • Passage Topics will follow the general specifications provided in the FCAT Reading

Test Item and Performance Task Specifications. All passages will be written specifically for this test. They will be engaging and high quality, free from bias and stereotyping, age-appropriate for the students, will present different points of view, and will include universal themes. The passages will also bring a range of diversity to the test, reflecting the variety of interests and backgrounds that make up Florida‘s student population. For example, characters will have names that reflect the diverse populations of Haitian-Creoles and Hispanics. Informational passages will provide accurate, fact-checked information. And, importantly, the passages will meet the needs of the Sunshine Standards.

• —Familiar stories“ is a phrase used in the Access Points. Since the passages are being written for the test, we will be sure the passages are about topics that are familiar to students at specific grade levels. For students in the elementary grades, the topics will relate to family or school life and opportunities students generally have in school. For students at the middle school grades, topics will also be familiar but will expand to more school-wide opportunities, outside the classroom. Students at the high school grades will see passages related to family, school and work transitions. Passages will be age-appropriate.

• The balance of Literary to Informational Texts will vary from grade to grade following this chart from page 3 of the FCAT Reading Test Item and Performance Task Specifications.

Grade Literary Text Informational Text

3 60% 40% 4 50% 50% 5 50% 50% 6 50% 50% 7 40% 60% 8 40% 60% 9 30% 70% 10 30% 70%

• Passage forms will follow the specifications from page 4 of the FCAT Reading Test Item and Performance Task Specifications.


Forms of Informational Text Forms of Literary Text • Content-area text (e.g., science, history) • Magazine and newspaper articles • Diaries • Editorials • Informational essays • Biographies and autobiographies • Primary Sources (e.g., Bill of Rights) • Consumer Materials • How-to articles • Advertisements • Tables and graphical presentations of

text (e.g., illustrations, photographs, and captions

• Short stories • Literary essays (e.g.,

critiques, personal narratives)

• Excerpts • Poems • Historical fiction • Fables and folk tales • Plays

• Graphics, both for passages and item responses, will be black and white line drawings or gray scale.

• Passages will include one to two graphics that set the scene/event of the story.

• Passage length will vary from the specifications for general education tests. Because of the needs of this particular population, the number of words in the passages will be about 25% less than the lowest range at a particular grade level. For example, at grade 3 the range of number of words is 100 -700 for the general education population. For this test, the range will be 75 -100 for grade 3.

Grade Range of Number of Words

3 75 - 100 4 75 - 100 5 150 - 200 6 150 - 200 7 225 - 300 8 225 - 300 9 225 - 300 10 225 - 300

• Passage Readabilities will vary. For grades 3, 4, and 5, the readabilities will not exceed grade 3 using the Spache Scale. For grades 6 through high school, the readability will range from 3 to 6 using Powers. Because we recognize that no readability formula is perfect, we rely on the Review Committee of Practitioners to help make the passages appropriate for the student population and yet to make the test an experience that measures what a student knows and is able to do.

A sample set of passages will submitted for approval by the DOE. The sample set will provide passages across the grade levels at the three access levels. This sample set will then be used as the basis for the development of all subsequent passages.


Writing

Design

The writing design consists of two strands that will be measured by the items in the test. In addition, at grades 8 and 10 two standards for each of the two strands will be identified for assessment. At grade 4 three standards will be assessed for the first strand and one standard for the second strand. Each standard will consist of two to eight items for a total of twenty-four writing items.

Blueprint

In developing the test blueprint for Writing, Measured Progress examined the same documents listed for reading and tried to follow the same methodology. We found the LA.3.5 standard (The student will write a final product for the intended audience) identified as an alternate in the Crosswalk documents at all grade levels. We know that students taking this test widely use application to learn, so writing applications would be consistent with their learning styles. Table 5a in the FCAT Summary of Tests and Design (September 2005) lists the modes for prompts for the writing portion of the test: narrative, expository, and persuasive. Finally, we found that the Philosophy for FCAT Writing + Assessment (2005) states, —The Best way to test student writing is to have students write.“

Therefore, we have included the Writing Application Strand for this test. A final product is specified in the Strand, Writing Applications. Therefore, in addition to the Writing Process Strand, we are including Writing Applications and focusing on narrative writing at grade 4 because this corresponds with general education student instructional learning at that grade level. In grade 8 we turn the focus to expository/ informational writing. For grade 10, the focus will be on expository/persuasive writing.

Grade Narrative Writing to tell a story

Expository Writing to explain

Persuasive Writing to convince

4 x 8 x x x 10 x x x

This means that for writing, overall, there are two strands assessed, Writing Process and Writing Applications, each with two standards. All grade levels are tested in Standard 4, Editing for Language Conventions. Writing Process Standards are tested at all levels, but the specific standard varies. Standard 1, Pre-Writing is not tested. It could be, but the FCAT emphasizes Drafting at grade 4 and Revising at grade 8. It makes sense to test Revising at Grade 10 also,

Rather than Prewriting. Writing Applications is tested at all levels, but the specific standard varies.

Strand 1 Writing Process: Grade 4

Grade 8

Grade 10


Standard 1: 8.3.1 Pre-Writing – The student will use prewriting strategies to generate ideas and formulate a plan. LA.910.3.1.1 - generating ideas from multiple sources (e.g., brainstorming, notes, journals, discussion, research materials or other reliable sources) based upon teacher-directed topics and personal interests …. LA.8.3.1.2 - making a plan for writing that addresses purpose, audience, main idea, logical sequence, and time frame for completion…. Standard 2: 4.3.2 Drafting – The student will write a draft appropriate to the topic, audience, and purpose. LA.4.3.2.1 - using a pre-writing plan to focus on the main idea with ample development of supporting details that shows an understanding of facts and/or opinions ….

4

LA.910.3.2.2 - establishing a logical organizational pattern with supporting details that are substantial, specific, and relevant …. LA.4.3.2.3 - creating interesting leads through the use of quotations, questions, or descriptions …. 4

Standard 3: 8.3.3 Revising -The student will revise and refine the draft for clarity and effectiveness. LA.8.3.3.1 - evaluating the draft for development of ideas and content, logical organization, voice, point of view, word choice, and sentence variation ….

4 4

LA.8.3.3.2 - creating clarity and logic by maintaining central theme, idea, or unifying point and developing relationships among ideas …. 4

LA.910.3.3.4 - applying appropriate tools or strategies to evaluate and refine the draft (e.g., peer review, checklists, rubrics). 4

Standard 4: 4.3.4 Editing for Language Conventions - The student will edit and correct the draft for the standard language conventions. LA.4.3.4.1 - spelling, using spelling rules, orthographic patterns, and generalizations (e.g., r-controlled, diphthong, consonant digraphs, vowel digraphs, silent e, plural for words ending in œy, doubling final consonant, i before e, irregular plurals, CVC words, CCVC words, CVCC words, affixes) and using a dictionary, thesaurus, or other resources as necessary; words used as names (e.g., Uncle Jim, Mom, Dad, Jr.) ….

2 2 1

LA.4.3.4.2 - capitalization for proper nouns, including titles used with someone‘s name, initials…. 1 1 1

LA.4.3.4.3 - punctuation, including end punctuation, apostrophes, commas, colons, quotation marks in dialogue, and apostrophes in singular possessives ….

1 1 2

LA.4.3.4.4 - present and past verb tense, noun-pronoun agreement, noun-verb agreement, subjective and objective pronouns, demonstrative pronouns and conjunctions;

1 1 2

LA.4.3.4.5 - subject/verb and noun/pronoun agreement in simple and compound sentences …. 1 1

Standard: 4.3.5 Publishing - The student will write a final product for the intended audience. LA.4.3.5.1 - prepare writing using technology in a format appropriate to audience and purpose (e.g., manuscript, multimedia)…. 2


Strand 2 Writing Applications: Grade 4

Grade 8

Grade 10

Standard: 4.4.1 Creative - The student develops and demonstrates creative writing. L.A. 4.4.1.1 - write narratives based on real or imagined ideas, events, or observations that include characters, setting, plot, sensory details, a logical sequence of events, and a context to enable the reader to imagine the world of the event or experience ….

8 4 4

Standard: 4.4.2 Informational -The student develops and demonstrates technical writing that provides information related to real-world tasks. LA.4.4.2.1 - write in a variety of informational/expository forms (e.g., summaries, procedures, recipes, instructions, graphs/tables, experiments, rubrics, how-to manuals)….

2

LA.4.4.2.2 - record information (e.g., observations, notes, lists, charts, map labels, legends) related to a topic, including visual aids as appropriate …. 2

LA.4.4.2.3 - write informational/expository essays that contain introductory, body, and concluding paragraphs …. 2

LA.910.4.2.4 - write a business letter and/or memo that presents information purposefully and succinctly to meet the needs of the intended audience following a conventional format (e.g., block, modified block, memo, email);

2

LA.910.4.2.5 - write detailed travel directions and design an accompanying graphic using the cardinal and ordinal directions, landmarks, streets and highways, and distances;…

2

LA.910.4.2.6 - write a work-related document (e.g., application, resume, meeting minutes, memo, cover letter, letter of application, speaker introduction, letter of recommendation)….

2

(The grade level number in the benchmark shows the grade level at which the benchmark is first tested; the benchmark may expand at higher grade levels; hence the ellipses. Numbers indicate how many item sets will be written at each grade level for each benchmark.)

Appendix G— Item Specifications Document 171 2007–08 Florida Alt Technical Report

Mathematics

Design The mathematics design will consist of 6 items from each of the three —Big Ideas“ and 6 items from —Supporting Ideas“ for grades 3-8 for a total of 24 items assessed. In grades 9 and 10 four Secondary Bodies of Knowledge will be assessed at each grade, with 4 to 8 items per Body of Knowledge for a total of 24 items.

Blueprint

Grades 3 through 8 For each of grades 3 through 8, the state‘s Mathematics Standards contain three —Big Ideas“ and three or more —Supporting Ideas.“ While these standards are still in draft form and, hence, not the basis for the current FCAT test blueprint, it is Measured Progress‘ assumption that the adoption of these standards will lead to a blueprint more like the one we propose here. It is also our assumption that sub-scores will be reported out by the —Big Ideas“ rather than by content strands, as done previously.

This actually makes it possible for the alternate assessment to more closely mirror the general assessment in both design and reporting. The —Big Ideas“ are sufficiently few in number and sufficiently broad in scope that is feasible to have a special education curriculum that encompasses all of them for each grade, based on the access points defined in the Mathematics Standards document.

As a result, our proposed test blueprint is for each grade‘s common assessment to contain: • 2 to 8 items coded to each the three Big Ideas • 6 items coded to the Supporting Ideas

Grade 3

Big Idea Number of Items IN Level SU Level PA Level

3.1 8 At least 1

each access point

4 each access point

4 each access point

3.2 2 2 each access point

2 each access point

1 each access point

At least 2 At least 2 At least 2 3.3 8 each access each access each access

point point point

Supporting Ideas 6

Alg: 2 G&M: 2 N&O: 1 DA: 1

Alg: 2 G&M: 2 N&O: 1 DA: 1

Alg: 2 G&M: 2 N&O: 1 DA: 1


Grade 4

Grade 5

Grade 6


4.1 6

a: 2 b: 1 c: 2 d: 1

a: 1 b: 2 c: 3

3 each access point


3 each access point

At least 1 each access

point

4.3 6


for a


for a


point

Supporting Ideas 6

Alg: 2 G&M: 2 N&O: 2

Alg: 2 G&M: 2 N&O: 2

Alg: 2 G&M: 2 N&O: 2



4 each access point

4 each access point


1 each access point

1 each access point

5.3 8 2 each, none

for b At least 1

each access At least 1

each access point point

Supporting Ideas 6

Alg: 2 G&M: 2 N&O: 1 DA: 1

Alg: 2 G&M: 2 N&O: 1 DA: 1

Alg: 2 G&M: 2 N&O: 1 DA: 1



point point point

6.2 6 a: 4 b: 2

a: 4 b: 2

2 each access point


point point point

Supporting Ideas 6

G&M: 2 N&O: 2 DA: 2

G&M: 2 N&O: 2 DA: 2

G&M: 2 N&O: 2 DA: 2


Grade 7

Grade 8


7.1 6 a: 4 b: 2


point

2 each access point


a: 2 b: 2 c: 1 d: 1

3 each access point


point point point G&M: 2 G&M: 2 G&M: 2

Supporting Ideas 6 N&O: 2

DA: 1 N&O: 2 DA: 1

N&O: 2 DA: 1

PR: 1 PR: 1 PR: 1


8.1 8 At least 2

each access point

a: 3 b: 1 c: 3 d: 1

4 each access point


point point point


1 each access point

2 at access point

Supporting Ideas 6

Alg: 2 G&M: 2 N&O: 2

Alg: 2 G&M: 2 N&O: 2

Alg: 2 G&M: 2 N&O: 2

Grades 9 and 10 The draft Content Standards are organized according to the following Secondary Bodies of Knowledge:

• Algebra • Geometry • Probability • Statistics • Finite Mathematics • Financial Literacy

Each body of knowledge is organized by a number of standards, and for each standard there are a set of access points given.

The proposed test design does presume an emphasis on Algebra and Geometry that is typical of the curriculum for these grades in most states, along with coverage of the four other bodies of knowledge.


Grade 9 • 8 items from the Algebra body of knowledge • 8 items from the Geometry body of knowledge • 4 items from the Financial Literacy of knowledge • 4 items from the Finite mathematics body of knowledge

Grade 10 • 8 items from the Algebra body of knowledge • 8 items from the Geometry body of knowledge • 4 items from the Financial Literacy body of knowledge • 2 items from the Probability body of knowledge • 2 items from the Statistics body of knowledge

Grade 9

Content Area Total number of items

Item Distribution By Standard

Algebra 8 A.1: 2 A.2: 2 A.3: 4

Geometry 8 G.1: 2 G.2: 2 G.3: 2 G.4: 2

Financial Literacy* 4 F.1: 1 F.2: 1 F.3: 2

Discrete Math 4 2 items to each bullet for each level

Grade 10

Content Area Total number of items

Item Distribution By Standard

Algebra 8

A.4: 2 A.5: 1 A.6: 1 A.7: 2 A.8: 2

Geometry 8 G.5: 2 G.6: 2 G.7: 2 G.8: 2

Financial Literacy* 4 F.1: 1 F.2: 1 F.3: 2

Probability 2 P.1: 1 P.2: 1

Statistics 2 S.3: 2 * Grades 9 and 10 will be distinguished by complexity of the items


Science

Design The Science design consists of the three major content areas in which scientific thinking/Standard 4 is embedded. Each of the major content areas will include 6 to 10 items with two of the items focusing on Standard 4. The test will consist of a total of 24 items.

Blueprint In developing the test blueprint for science, several documents were examined:

• Florida Alternate Assessment Report Resource Manual, Revised 2006 • Performance Objectives for Elementary Students • Alternate Assessment in Science for Students with Disabilities • Grade Level Expectations for the SSS, Science Grades PreK-2 • Grade Level Expectations for the SSS, Science Grades 3-5 • Grade Level Expectations for the SSS, Science Grades 6-8 • Sunshine State Standards, Science Grades 9-12

The benchmarks for which students must be assessed in Science Alternate Assessment all fall within one standard–Standard 4: The student uses systematic approaches when solving problems. However, science encompasses a wide range of topics within three major content areas. According to the Alternate Assessment in Science for Students with Disabilities, 2004, the following guidelines should be used in determining which alternate assessments should be used to access science.

• The skills assessed in alternate assessment should reflect the use of the problem solving process required in Functional Academics, Standard 4: The student uses systematic approaches when solving problems.

• The content assessed in alternate assessment should generally reflect the same areas assessed by the FCAT: physical and chemical sciences, earth and space sciences, life and environmental sciences, and scientific thinking.

In order to meet the above criteria, the blueprint distributes the assessment items across the science content areas covered in FCAT as well as the Functional Academics Standard 4. Therefore, the Science Blueprint chart involves:

1. Distribution of Standard 4 benchmarks across each grade level. 2. Distribution of major science content strands across each grade level.

To achieve this two-fold distribution, each assessment item will reflect a science content strand and within each science content area, two items will also assess an appropriate Standard 4 Problem Solving Benchmark. Thus, Standard 4 will be embedded in the science content. Items will focus on the science content assessed by the FCAT at each grade level. When the on-grade-level content is abstract or too complex for the intended population, the items will backtrack to the corresponding benchmark at a lower grade level in order to assess the prerequisite content.


An emphasis is placed on the FCAT strands most appropriate for alternate assessment. For example within the content area of Physical and Chemical Sciences, the strand, The Nature of Matter, is emphasized because it deals with observable and measurable properties of matter such as common materials and simple measurements. Less emphasis is placed on the strands dealing with the concepts of energy, force and motion, because they are inherently abstract concepts.

In Earth and Space Sciences, Strand D, Processes that Shape the Earth, will have greater coverage because these processes include more concrete topics such as rocks, water, weather, types of soil, recycling, and conservation. The topics within Strand E, Earth and Space, include less concrete topics such as lunar cycle, differences between celestial bodies, and the vastness of the universe.

In Life and Environmental Sciences, Strand F, Processes of Life, is given greater coverage because this content deals with functions of the body systems, survival of organisms, and similarities and differences of plants and animals.

The remaining content area, Scientific Thinking, has only one strand, which corresponds with Standard 4 and it is distributed evenly across the three grade levels.

Distribution Across Science Content Areas Grade 5 Grade 8 Grade 11

Physical and Chemical Sciences 8 8 8 Strand A: The Nature of Matter 4 4 4 Strand B: Energy 2 2 2 Strand C: Force and Motion 2 2 2 Standard 4: The student uses systematic approaches to solve problems* 2* 2* 2*

Earth and Space Sciences 6 6 6 Strand D: Process that Shape the Earth 4 4 4 Strand E: Earth and Space 2 2 2 Standard 4: The student uses systematic approaches to solve problems* 2* 2* 2*

Life and Environmental Sciences 10 10 10 Strand F: Process of Life 6 6 6 Strand G: How Living Things Interact with Their Environment 4 4 4 Standard 4: The student uses systematic approaches to solve problems* 2* 2* 2*

*Items that assess Standard 4 (Scientific Thinking) will also assess a science content benchmark.


Item Specifications

• Items should clearly address the concept and/or skill described in the access points for each level. To the extent possible, the tasks for each of the access points within a given item should be related, i.e., the task for the Independent access point should assess the same concept and/or skill as the task for the Participatory, but at a higher level of cognitive demand.

• Where students are asked to select a single choice from a set of options, there should be at most three options provided. On occasion, however, for example in an item that asks a student to recognize examples and non-examples of a given concept, students may be given up to six options and asked to address each one. (e.g., show six different shapes and ask student to identify all the ones that are squares)

• Passages will be read aloud to the student.

• At all Access Points, word cards and sentence strips will be read to the student.

• Where not otherwise specified in the standard being assessed, numbers and other elements of items should be kept as simple as possible.

• To determine whether a word is appropriate to use in an item, a variety of sources will be used: Dolch Basic Sight Word List, Revised Dolch List, the work of Chall and Popp described in Teaching and Assessing Phonics: Why, What, When, How (Educators Publishing Service, Inc. 1996), EDL Core Vocabularies in Reading, Mathematics, Science, and Social Studies,( Steck-Vaughn Company, 1989), and The Living Word by Dale and O‘Rourke (World Book-Childcraft International, Inc., 1981). Again, we will rely on the Review Committee of Practitioners to help make the word choices appropriate for the student population and make the test an experience that measures what a student knows and is able to do.

• Simple content terminology will be used in grades 3-5, with more accurate content terminology usage at grades 6-11. For example in grades 3-5 the question may be —What is the story mostly about?“ and at grades 6-11 the question will be —What is the main idea?.“

• To the extent possible, items should involve situations or contexts that can be expected to be familiar to most students and that are age-appropriate. In particular, items for the secondary grades should involve situations, contexts, and objects that are of interest to older students.

• Items may presume the use of some readily available classroom materials, such as counters. However, most items should include all necessary materials, e.g., shapes. Shapes and other manipulatives (e.g., picture cards) will be provided as graphics on regular paper that classroom teachers will cut out for the students‘ use on the assessment. Graphics of objects that may be replaced by the real object need to be small enough to fit on a desk space and to remain stable (not rolling around).

• Participatory Access Point Students at the Participatory Access Point may respond indicating the answer with the mode the student most commonly uses in class to communicate, such as yes/no cards, picture cards, eye gaze, assistive technology, and signing. Materials provided for


responses will be picture cards, and number cards. The key and the two distracters will be clearly different from each other.

• Supported Access Point Students at the Supported Access Point may respond indicating the answer with the mode the student most commonly uses in class to communicate, such as picture, word cards, sentence strips, eye gaze, assistive technology, and signing. Materials provided for responses will be picture cards, word cards (with pictures for more difficult words), sentence strips (with pictures) and number cards. At least two of the options, the key and one distracter, will relate to the item stimulus.

• Independent Access Point Students at the Independent Access Point may respond indicating the answer with the mode the student most commonly uses in class to communicate, such as sentence strips, verbal or written responses, eye gaze, assistive technology, and signing. Materials provided for responses will be picture cards, word cards, sentence strips and number cards. All options, the key and the distracters, will relate to the item stimulus, or the student may be expected to provide the answer.

• Other item specifications will follow two sets of guidelines: 1. Those described in the FCAT Reading, Writing, Mathematics and Science Test Item and

Performance Task Specifications. 2. Item-writing guidelines typically followed by Measured Progress.

a. Aligned to the particular standard and appropriate level of difficulty, b. Items and tasks are clear, concise and easy to read. c. Having one and only one answer for multiple choice, d. Irrelevant clues to the correct answer are avoided. e. Most items will be positively worded. f. Distracters will have similar length. g. All options will be similar in grammatical structure and form. h. Item context will avoid any cultural, racial, or gender bias. i. Items will follow the principles of Universal Design


APPENDIX H—RAW SCORE CUMULATIVE DISTRIBUTIONS

Appendix H—Raw Score Cumulative Dist. 181 2007–08 Florida Alt Technical Report

182










192

APPENDIX I—ITEM ANALYSIS RESULTS

Appendix I— Item Analysis Results 193 2007-08 Florida ALT Technical Report


Table I-1a. 2007–08 Florida Alternate Assessment: Rubric Score Mean and S.D., Item Difficulty, and Discrimination

by Grade and Content Area – Mathematics (All Students) Content Item Mean Rubric Grade N s.d. Difficulty Discrimination Area Position Score

1 2061 3.91 2.81 0.43 0.72 2 2074 5.96 3.41 0.66 0.79 3 2060 3.40 3.00 0.38 0.48 4 2077 4.96 3.24 0.55 0.78 5 2082 5.02 3.07 0.56 0.69 6 2074 3.70 2.26 0.41 0.70 7 2076 5.94 3.46 0.66 0.79

Mathematics 3 8 9

2044 2028

3.85 4.24

3.09 3.48

0.43 0.47

0.73 0.69

10 2050 4.48 3.49 0.50 0.65 11 2055 4.37 2.77 0.49 0.76 12 2054 4.51 3.25 0.50 0.76 13 2054 3.77 2.60 0.42 0.65 14 2037 4.29 3.17 0.48 0.75 15 2052 4.11 3.02 0.46 0.73 16 2044 2.48 2.27 0.28 0.52 1 2147 3.87 2.44 0.43 0.72 2 2138 4.50 2.88 0.50 0.81 3 2127 4.10 3.10 0.46 0.75 4 2129 3.36 3.20 0.37 0.57 5 2146 5.32 3.19 0.59 0.78 6 2119 4.47 3.06 0.50 0.78 7 2132 4.36 2.61 0.48 0.75

Mathematics 4 8 9

2137 2144

3.84 4.69

2.55 2.90

0.43 0.52

0.67 0.71

10 2143 4.30 2.80 0.48 0.70 11 2144 5.25 3.37 0.58 0.78 12 2128 4.89 3.03 0.54 0.76 13 2134 4.81 3.01 0.53 0.80 14 2137 4.03 2.22 0.45 0.71 15 2124 5.37 3.38 0.60 0.77 16 2133 5.56 3.29 0.62 0.79 1 2238 3.38 2.76 0.38 0.68 2 2237 3.53 2.96 0.39 0.72 3 2249 5.34 3.34 0.59 0.77 4 2247 4.75 3.03 0.53 0.77 5 2235 4.57 3.03 0.51 0.75 6 2236 4.71 3.26 0.52 0.76 7 2210 4.28 3.33 0.48 0.77

Mathematics 5 8 9

2236 2223

4.36 4.08

2.86 3.14

0.48 0.45

0.80 0.76

10 2239 4.96 3.22 0.55 0.77 11 2239 3.78 2.60 0.42 0.76 12 2243 4.07 2.52 0.45 0.68 13 2221 4.38 3.14 0.49 0.77 14 2204 4.80 3.52 0.53 0.77 15 2218 3.93 2.97 0.44 0.70 16 2227 4.88 2.83 0.54 0.77

(cont‘d)


Content

Area Grade Item Position N Mean Rubric

Score s.d. Difficulty Discrimination

1 2183 5.53 3.36 0.61 0.78 2 2175 5.92 3.38 0.66 0.81 3 2179 4.71 2.92 0.52 0.78 4 2180 5.89 3.35 0.65 0.80 5 2171 4.76 2.65 0.53 0.77 6 2137 4.05 3.20 0.45 0.69 7 2142 3.80 2.88 0.42 0.71

Mathematics 6 8 9

2163 2172

3.86 5.41

2.91 3.40

0.43 0.60

0.66 0.82

10 2173 4.55 3.15 0.51 0.73 11 2189 4.86 3.12 0.54 0.75 12 2165 3.87 2.52 0.43 0.67 13 2148 4.04 3.26 0.45 0.75 14 2160 4.51 3.33 0.50 0.72 15 2177 4.83 2.88 0.54 0.71 16 2135 3.91 3.16 0.43 0.71 1 2361 4.57 2.76 0.51 0.61 2 2327 5.40 3.60 0.60 0.78 3 2319 4.07 2.70 0.45 0.69 4 2340 4.66 2.70 0.52 0.62 5 2340 5.44 3.31 0.60 0.81 6 2330 4.34 2.93 0.48 0.75 7 2321 4.60 3.36 0.51 0.76

Mathematics 7 8 9

2323 2325

4.67 4.75

3.18 2.93

0.52 0.53

0.74 0.71

10 2325 4.36 2.81 0.48 0.78 11 2324 4.57 2.80 0.51 0.73 12 2316 4.37 3.25 0.49 0.73 13 2338 4.50 3.01 0.50 0.70 14 2338 5.00 2.73 0.56 0.75 15 2315 4.84 3.13 0.54 0.78 16 2336 5.06 2.66 0.56 0.75 1 2375 5.99 3.28 0.67 0.77 2 2353 4.97 3.42 0.55 0.77 3 2383 4.29 2.78 0.48 0.71 4 2374 5.02 2.73 0.56 0.73 5 2369 3.87 2.68 0.43 0.64 6 2372 4.54 2.51 0.50 0.76 7 2370 4.93 2.72 0.55 0.76

Mathematics 8 8 9

2372 2359

4.94 3.83

3.00 2.83

0.55 0.43

0.75 0.68

10 2370 3.74 2.24 0.42 0.61 11 2379 4.50 2.92 0.50 0.71 12 2362 4.09 2.77 0.45 0.74 13 2381 4.87 2.99 0.54 0.66 14 2364 4.28 2.82 0.48 0.76 15 2372 4.26 2.71 0.47 0.75 16 2356 5.15 3.18 0.57 0.79

(cont‘d)


Content Area Grade Item

Position N Mean Rubric Score s.d. Difficulty Discrimination

1 2839 5.46 3.04 0.61 0.65 2 2820 5.36 3.48 0.60 0.81 3 2836 4.07 2.56 0.45 0.55 4 2833 4.27 2.90 0.47 0.71 5 2827 5.20 3.19 0.58 0.75 6 2827 4.29 2.95 0.48 0.62 7 2812 4.48 3.03 0.50 0.65

Mathematics 9 8 9

2838 2827

5.27 5.35

3.26 3.42

0.59 0.59

0.80 0.80

10 2821 5.51 3.15 0.61 0.72 11 2816 4.37 3.18 0.49 0.72 12 2842 5.57 2.90 0.62 0.77 13 2817 5.27 3.05 0.59 0.80 14 2815 5.04 3.51 0.56 0.77 15 2810 4.98 3.32 0.55 0.76 16 2817 6.65 3.23 0.74 0.78

Mathematics 10

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16

2941 2939 2920 2935 2928 2930 2938 2950 2939 2932 2936 2933 2919 2940 2933 2896

4.86 5.56 3.97 4.50 4.96 6.38 4.93 5.45 5.27 4.08 5.51 4.68 4.59 5.64 4.47 4.71

2.79 2.97 2.54 2.56 3.03 3.28 3.07 2.86 2.53 2.59 3.24 2.69 3.12 2.86 2.88 3.07

0.54 0.62 0.44 0.50 0.55 0.71 0.55 0.61 0.59 0.45 0.61 0.52 0.51 0.63 0.50 0.52

0.64 0.76 0.58 0.73 0.78 0.81 0.70 0.78 0.74 0.65 0.77 0.70 0.66 0.80 0.67 0.71


Table I-1b. 2007–08 Florida Alternate Assessment: Rubric Score Mean and S.D., Item Difficulty, and Discrimination by

Grade and Content Area – Mathematics (Participatory Students) Content Item Mean Rubric Grade N s.d. Difficulty Discrimination Area Position Score

1 839 1.60 1.44 0.18 0.47 2 839 2.56 2.47 0.28 0.59 3 839 1.89 1.76 0.21 0.52 4 839 1.89 1.50 0.21 0.51 5 839 2.62 2.32 0.29 0.56 6 839 2.02 1.47 0.22 0.64 7 839 2.40 2.37 0.27 0.56

Mathematics 3 8 9

839 839

1.47 1.57

1.35 1.54

0.16 0.17

0.44 0.51

10 839 1.92 1.88 0.21 0.48 11 839 2.07 1.72 0.23 0.62 12 839 1.73 1.65 0.19 0.49 13 839 1.94 1.69 0.22 0.58 14 839 1.53 1.35 0.17 0.56 15 839 1.75 1.38 0.19 0.60 16 839 1.47 1.13 0.16 0.50 1 787 1.94 1.39 0.22 0.64 2 787 1.77 1.34 0.20 0.60 3 787 1.70 1.29 0.19 0.52 4 787 1.54 1.34 0.17 0.37 5 787 2.26 1.90 0.25 0.56 6 787 1.72 1.36 0.19 0.57 7 787 2.05 1.54 0.23 0.63

Mathematics 4 8 9

787 787

1.99 2.28

1.47 1.81

0.22 0.25

0.66 0.65

10 787 2.13 1.69 0.24 0.60 11 787 2.03 1.68 0.23 0.55 12 787 2.06 1.82 0.23 0.58 13 787 2.08 1.68 0.23 0.58 14 787 2.16 1.67 0.24 0.63 15 787 1.97 1.91 0.22 0.49 16 787 2.22 1.84 0.25 0.62 1 1007 1.77 1.17 0.20 0.46 2 1007 1.55 1.24 0.17 0.39 3 1007 2.48 2.09 0.28 0.50 4 1007 2.42 1.89 0.27 0.56 5 1007 2.16 1.87 0.24 0.49 6 1007 2.10 1.60 0.23 0.56 7 1007 1.73 1.32 0.19 0.51

Mathematics 5 8 9

1007 1007

2.10 1.72

1.42 1.29

0.23 0.19

0.63 0.53

10 1007 2.40 1.91 0.27 0.56 11 1007 1.96 1.17 0.22 0.68 12 1007 2.38 1.75 0.26 0.59 13 1007 1.86 1.62 0.21 0.46 14 1007 1.85 1.59 0.21 0.47 15 1007 2.00 1.39 0.22 0.56 16 1007 2.58 2.13 0.29 0.63

(cont‘d)




1 838 2.30 2.13 0.26 0.48 2 838 2.34 2.17 0.26 0.55 3 838 2.15 1.61 0.24 0.57 4 838 2.53 2.03 0.28 0.61 5 838 2.28 1.89 0.25 0.58 6 838 1.65 1.47 0.18 0.52 7 838 1.68 1.31 0.19 0.58

Mathematics 6 8 9

838 838

1.65 1.93

1.26 1.68

0.18 0.21

0.54 0.57

10 838 1.89 1.71 0.21 0.43 11 838 2.16 1.87 0.24 0.51 12 838 1.98 1.46 0.22 0.61 13 838 1.35 1.12 0.15 0.43 14 838 1.66 1.68 0.18 0.41 15 838 2.43 1.98 0.27 0.60 16 838 1.59 1.25 0.18 0.55 1 819 2.51 2.01 0.28 0.60 2 819 1.78 1.47 0.20 0.45 3 819 1.83 1.38 0.20 0.60 4 819 2.56 2.04 0.28 0.66 5 819 2.09 1.59 0.23 0.62 6 819 1.76 1.40 0.20 0.60 7 819 1.68 1.31 0.19 0.53

Mathematics 7 8 9

819 819

1.93 2.21

1.50 1.78

0.21 0.25

0.66 0.62

10 819 1.79 1.40 0.20 0.60 11 819 1.96 1.57 0.22 0.65 12 819 1.65 1.49 0.18 0.50 13 819 2.05 1.60 0.23 0.62 14 819 2.31 1.81 0.26 0.69 15 819 1.85 1.54 0.21 0.57 16 819 2.37 1.81 0.26 0.68 1 765 2.26 2.01 0.25 0.58 2 765 1.62 1.31 0.18 0.58 3 765 1.91 1.40 0.21 0.66 4 765 2.09 1.79 0.23 0.63 5 765 1.70 1.45 0.19 0.55 6 765 2.11 1.68 0.23 0.67 7 765 2.16 1.78 0.24 0.65

Mathematics 8 8 9

765 765

1.97 1.64

1.50 1.28

0.22 0.18

0.66 0.59

10 765 2.08 1.54 0.23 0.69 11 765 2.02 1.64 0.22 0.64 12 765 1.56 1.28 0.17 0.56 13 765 2.28 1.80 0.25 0.63 14 765 1.61 1.26 0.18 0.58 15 765 1.83 1.37 0.20 0.66 16 765 1.78 1.59 0.20 0.57

(cont‘d)




Mathematics 9

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16

872 872 872 872 872 872 872 872 872 872 872 872 872 872 872 872

2.80 1.60 2.25 1.63 2.18 2.15 1.98 1.88 1.75 2.32 1.71 2.40 2.01 1.54 1.73 2.65

2.24 1.34 1.94 1.32 1.68 1.76 1.78 1.49 1.32 2.08 1.50 1.81 1.64 1.27 1.34 2.44

0.31 0.18 0.25 0.18 0.24 0.24 0.22 0.21 0.19 0.26 0.19 0.27 0.22 0.17 0.19 0.29

0.60 0.50 0.52 0.37 0.56 0.55 0.46 0.50 0.48 0.56 0.48 0.66 0.56 0.45 0.55 0.59

Mathematics 10

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16

848 848 848 848 848 848 848 848 848 848 848 848 848 848 848 848

1.97 2.23 1.94 1.84 2.48 2.22 2.01 2.15 1.85 2.22 1.98 2.41 2.52 1.99 2.02 2.14

1.57 1.76 1.38 1.64 1.78 1.69 1.66 1.60 1.31 1.80 1.50 1.65 1.80 1.37 1.77 1.68

0.22 0.25 0.22 0.20 0.28 0.25 0.22 0.24 0.21 0.25 0.22 0.27 0.28 0.22 0.22 0.24

0.45 0.55 0.57 0.49 0.60 0.57 0.54 0.60 0.54 0.56 0.56 0.61 0.63 0.62 0.45 0.58


Table I-1c. 2007–08 Florida Alternate Assessment: Rubric Score Mean and S.D., Item Difficulty, and Discrimination by

Grade and Content Area – Mathematics (Supported Students) Content Item Mean Rubric Grade N s.d. Difficulty Discrimination Area Position Score

Mathematics 3

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

831 831 831 831 831 831 831 831 831 831 831 831 831 831 831 831

4.52 7.61 3.57 6.14 5.76 4.06 7.76 4.08 4.57 5.07 4.95 5.13 4.27 5.05 4.44 2.14

2.49 2.18 2.88 2.63 2.50 1.72 2.14 2.76 3.24 3.31 2.14 2.73 2.15 2.65 2.46 1.51

0.50 0.85 0.40 0.68 0.64 0.45 0.86 0.45 0.51 0.56 0.55 0.57 0.47 0.56 0.49 0.24

0.13 0.11 -0.13 0.17 -0.02 0.05 0.13 0.20 0.00 0.03 0.17 0.12 0.03 0.13 0.15 0.02

Mathematics 4

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

800 800 800 800 800 800 800 800 800 800 800 800 800 800 800 800

3.90 4.87 3.65 2.66 5.92 4.69 4.73 3.89 4.99 4.39 5.77 5.54 5.01 4.47 6.32 6.47

1.79 2.27 2.35 2.54 2.50 2.49 2.03 1.83 2.20 2.24 2.88 2.32 2.26 1.81 2.71 2.52

0.43 0.54 0.41 0.30 0.66 0.52 0.53 0.43 0.55 0.49 0.64 0.62 0.56 0.50 0.70 0.72

0.04 0.32 0.19 -0.01 0.18 0.26 0.13 -0.03 -0.05 0.03 0.20 0.14 0.22 0.10 0.18 0.16

1 681 3.05 2.13 0.34 0.13 2 681 3.27 2.42 0.36 0.13 3 681 6.66 2.66 0.74 0.08 4 681 5.28 2.34 0.59 0.07 5 681 5.43 2.46 0.60 0.07 6 681 5.45 2.83 0.61 0.09 7 681 4.50 2.91 0.50 0.14

Mathematics 5 8 9

681 681

4.78 4.33

2.13 2.57

0.53 0.48

0.15 0.12

10 681 5.71 2.68 0.63 0.07 11 681 3.85 1.79 0.43 0.18 12 681 4.50 1.88 0.50 -0.07 13 681 5.05 2.60 0.56 0.15 14 681 5.80 3.17 0.64 0.08 15 681 3.91 2.47 0.43 0.05 16 681 5.95 1.80 0.66 0.03




1 697 6.31 2.68 0.70 0.13 2 697 7.27 2.31 0.81 0.13 3 697 4.84 2.05 0.54 0.10 4 697 6.85 2.70 0.76 0.11 5 697 5.46 1.74 0.61 0.13 6 697 3.83 2.55 0.43 0.06 7 697 3.46 2.15 0.38 0.01

Mathematics 6 8 9

697 697

4.04 6.32

2.52 2.66

0.45 0.70

0.02 0.20

10 697 4.79 2.77 0.53 0.11 11 697 5.11 2.37 0.57 0.05 12 697 4.05 1.92 0.45 0.07 13 697 3.70 2.62 0.41 0.13 14 697 4.80 2.84 0.53 0.06 15 697 5.26 2.32 0.58 0.04 16 697 3.29 2.23 0.37 0.05 1 859 4.92 2.23 0.55 -0.12 2 859 5.79 3.31 0.64 0.27 3 859 4.24 2.08 0.47 0.04 4 859 5.13 2.12 0.57 -0.11 5 859 5.75 2.75 0.64 0.27 6 859 4.30 2.27 0.48 0.11 7 859 4.28 2.77 0.48 0.22

Mathematics 7 8 9

859 859

4.62 5.04

2.53 2.43

0.51 0.56

0.17 0.10

10 859 4.39 2.08 0.49 0.13 11 859 5.03 2.23 0.56 0.04 12 859 4.11 2.62 0.46 0.10 13 859 4.51 2.40 0.50 0.02 14 859 5.75 2.07 0.64 0.12 15 859 5.04 2.51 0.56 0.18 16 859 5.92 1.99 0.66 0.17 1 1066 7.03 2.44 0.78 0.13 2 1066 5.23 3.01 0.58 0.30 3 1066 4.31 2.06 0.48 0.00 4 1066 5.89 1.98 0.65 0.07 5 1066 4.09 2.08 0.45 0.03 6 1066 4.92 1.86 0.55 0.18 7 1066 5.48 1.95 0.61 0.10

Mathematics 8 8 9

1066 1066

5.34 3.62

2.36 2.15

0.59 0.40

0.13 0.06

10 1066 3.90 1.69 0.43 -0.01 11 1066 4.48 2.22 0.50 0.04 12 1066 4.23 2.25 0.47 0.23 13 1066 5.21 2.56 0.58 0.05 14 1066 4.42 2.20 0.49 0.23 15 1066 4.30 2.07 0.48 0.17 16 1066 5.71 2.47 0.63 0.23

(cont‘d)




Mathematics 9

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16

914 914 914 914 914 914 914 914 914 914 914 914 914 914 914 914

5.50 5.07 4.07 4.01 4.75 3.86 4.21 4.91 4.93 5.69 3.46 5.93 5.20 4.50 4.47 7.74

2.37 3.00 2.18 2.25 2.58 2.23 2.36 2.51 2.86 2.51 2.36 2.14 2.38 3.03 2.66 2.16

0.61 0.56 0.45 0.45 0.53 0.43 0.47 0.55 0.55 0.63 0.38 0.66 0.58 0.50 0.50 0.86

-0.12 0.24 -0.18 0.15 0.12 -0.15 -0.04 0.17 0.19 0.08 0.07 0.12 0.21 0.16 0.15 0.14

Mathematics 10

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16

1128 1128 1128 1128 1128 1128 1128 1128 1128 1128 1128 1128 1128 1128 1128 1128

4.15 6.02 4.22 4.37 4.76 5.86 3.98 4.29 4.66 7.16 4.88 5.42 5.63 3.81 5.54 4.65

2.70 1.95 2.22 2.54 2.23 2.22 2.01 1.81 2.37 2.55 2.55 2.22 1.83 1.91 2.67 2.11

0.46 0.67 0.47 0.49 0.53 0.65 0.44 0.48 0.52 0.80 0.54 0.60 0.63 0.42 0.62 0.52

0.05 0.18 0.06 0.12 0.00 0.12 -0.04 0.14 0.23 0.33 0.10 0.27 0.16 0.01 0.26 0.07


Table I-1d. 2007–08 Florida Alternate Assessment: Rubric Score Mean and S.D., Item Difficulty, and Discrimination by

Grade and Content Area – Mathematics (Independent Students) Content Item Mean Rubric Grade N s.d. Difficulty Discrimination Area Position Score

1 521 6.63 1.82 0.74 0.14 2 521 8.79 0.82 0.98 0.05 3 521 5.58 3.34 0.62 0.17 4 521 8.04 1.64 0.89 0.10 5 521 7.71 1.88 0.86 0.11 6 521 5.83 1.98 0.65 0.21 7 521 8.74 0.87 0.97 0.00

Mathematics 3 8 9

521 521

7.30 7.98

1.97 2.19

0.81 0.89

0.18 0.07

10 521 7.69 2.61 0.85 0.07 11 521 7.13 1.91 0.79 0.24 12 521 7.98 1.77 0.89 0.15 13 521 5.94 2.46 0.66 0.28 14 521 7.53 2.12 0.84 0.15 15 521 7.39 2.39 0.82 0.10 16 521 4.65 3.09 0.52 0.24 1 650 6.16 2.12 0.68 0.28 2 650 7.34 1.66 0.82 0.21 3 650 7.58 2.21 0.84 0.27 4 650 6.41 3.36 0.71 0.09 5 650 8.30 1.47 0.92 0.09 6 650 7.55 1.93 0.84 0.10 7 650 6.69 1.86 0.74 0.30

Mathematics 4 8 9

650 650

6.03 7.24

2.59 2.34

0.67 0.80

0.30 0.20

10 650 6.82 2.29 0.76 0.31 11 650 8.51 1.42 0.95 0.08 12 650 7.53 1.94 0.84 0.22 13 650 7.87 1.73 0.87 0.23 14 650 5.77 1.45 0.64 0.22 15 650 8.30 1.48 0.92 0.10 16 650 8.50 1.48 0.94 0.08 1 651 6.23 2.88 0.69 0.32 2 651 6.85 2.42 0.76 0.20 3 651 8.38 1.49 0.93 0.11 4 651 7.80 1.92 0.87 0.18 5 651 7.41 1.83 0.82 0.24 6 651 7.98 2.03 0.89 0.13 7 651 7.97 2.15 0.89 0.15

Mathematics 5 8 9

651 651

7.42 7.45

2.03 2.44

0.82 0.83

0.23 0.21

10 651 8.14 1.80 0.90 0.14 11 651 6.53 2.48 0.73 0.27 12 651 6.23 2.28 0.69 0.29 13 651 7.58 1.92 0.84 0.20 14 651 8.33 1.76 0.93 0.12 15 651 6.92 2.76 0.77 0.28 16 651 7.33 1.74 0.81 0.28

(cont‘d)



Position N Mean Rubric Score

s.d. Difficulty Discrimination

1 753 8.42 1.42 0.94 0.07 2 753 8.64 1.10 0.96 0.11 3 753 7.46 2.04 0.83 0.25 4 753 8.74 1.13 0.97 0.06 5 753 6.88 1.66 0.76 0.21 6 753 6.92 2.81 0.77 0.10 7 753 6.48 2.59 0.72 0.34

Mathematics 6 8 9

753 753

6.15 8.42

2.74 1.49

0.68 0.94

0.18 0.12

10 753 7.29 2.09 0.81 0.20 11 753 7.63 2.16 0.85 0.25 12 753 5.81 2.38 0.65 0.16 13 753 7.33 2.35 0.81 0.23 14 753 7.41 2.35 0.82 0.08 15 753 7.09 2.02 0.79 0.11 16 753 7.05 2.78 0.78 0.24 1 770 6.38 2.54 0.71 0.27 2 770 8.80 1.03 0.98 0.02 3 770 6.25 2.51 0.69 0.31 4 770 6.35 2.44 0.71 0.24 5 770 8.65 1.20 0.96 0.08 6 770 7.14 2.15 0.79 0.24 7 770 8.08 2.02 0.90 0.15

Mathematics 7 8 9

770 770

7.64 7.14

2.42 2.17

0.85 0.79

0.15 0.21

10 770 7.06 1.98 0.78 0.31 11 770 6.84 2.04 0.76 0.21 12 770 7.53 2.36 0.84 0.24 13 770 7.11 2.54 0.79 0.25 14 770 7.01 1.73 0.78 0.21 15 770 7.78 1.85 0.86 0.23 16 770 6.95 1.61 0.77 0.18 1 655 8.66 1.10 0.96 0.04 2 655 8.47 1.51 0.94 0.10 3 655 7.05 2.40 0.78 0.27 4 655 7.04 1.66 0.78 0.07 5 655 6.04 2.73 0.67 0.32 6 655 6.75 1.74 0.75 0.31 7 655 7.27 1.82 0.81 0.27

Mathematics 8 8 9

655 655

7.77 6.70

1.97 2.64

0.86 0.74

0.15 0.28

10 655 5.41 2.38 0.60 0.26 11 655 7.43 2.36 0.83 0.21 12 655 6.82 2.04 0.76 0.27 13 655 7.33 2.34 0.81 0.11 14 655 7.17 1.93 0.80 0.26 15 655 7.03 2.05 0.78 0.24 16 655 8.18 1.55 0.91 0.13

(cont‘d)




Mathematics 9

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16

1155 1155 1155 1155 1155 1155 1155 1155 1155 1155 1155 1155 1155 1155 1155 1155

7.44 8.44 5.45 6.47 7.83 6.25 6.57 8.12 8.39 7.78 7.10 7.67 7.79 8.10 7.83 8.82

2.46 1.49 2.39 2.44 2.15 2.91 2.74 1.87 1.69 2.05 2.49 1.80 1.68 1.98 2.22 0.82

0.83 0.94 0.61 0.72 0.87 0.69 0.73 0.90 0.93 0.86 0.79 0.85 0.87 0.90 0.87 0.98

0.21 0.15 0.32 0.32 0.29 0.39 0.29 0.25 0.18 0.11 0.32 0.18 0.23 0.20 0.19 0.07

Mathematics 10

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16

1060 1060 1060 1060 1060 1060 1060 1060 1060 1060 1060 1060 1060 1060 1060 1060

7.14 7.96 6.76 7.36 6.87 7.92 5.54 6.61 7.75 8.88 7.35 7.91 7.10 6.05 8.27 6.72

2.45 1.46 2.57 2.06 2.41 1.76 2.54 2.08 1.82 0.69 2.35 1.59 1.59 2.52 1.61 2.09

0.79 0.88 0.75 0.82 0.76 0.88 0.62 0.73 0.86 0.99 0.82 0.88 0.79 0.67 0.92 0.75

0.21 0.23 0.22 0.18 0.19 0.17 0.26 0.37 0.26 0.04 0.17 0.16 0.15 0.27 0.13 0.26


Table I-2a. 2007–08 Florida Alternate Assessment: Rubric Score Mean and S.D., Item Difficulty, and Discrimination by

Grade and Content Area – Reading (All Students) Content Item Mean Rubric Grade N s.d. Difficulty Discrimination Area Position Score

1 2106 4.76 3.09 0.53 0.73 2 2091 4.21 2.93 0.47 0.72 3 2101 5.45 3.23 0.61 0.83 4 2093 4.82 3.10 0.54 0.80 5 2100 4.94 3.28 0.55 0.81 6 2098 4.31 3.14 0.48 0.69 7 2093 5.15 3.39 0.57 0.81

Reading 3 8 9

2090 2097

4.47 5.30

3.00 3.43

0.50 0.59

0.80 0.84

10 2091 5.58 3.09 0.62 0.74 11 2084 3.83 2.78 0.43 0.65 12 2095 4.76 2.96 0.53 0.77 13 2092 5.33 3.37 0.59 0.83 14 2071 5.39 3.46 0.60 0.82 15 2081 4.38 3.20 0.49 0.69 16 2080 5.00 3.13 0.56 0.84 1 2167 4.85 2.88 0.54 0.72 2 2154 4.43 2.88 0.49 0.69 3 2169 5.24 2.86 0.58 0.77 4 2140 4.59 3.36 0.51 0.75 5 2138 3.80 2.86 0.42 0.65 6 2154 4.62 3.20 0.51 0.76 7 2148 4.21 3.10 0.47 0.72

Reading 4 8 9

2165 2153

4.80 4.75

3.11 3.09

0.53 0.53

0.74 0.76

10 2157 3.88 2.84 0.43 0.64 11 2144 4.60 3.31 0.51 0.74 12 2130 3.54 2.69 0.39 0.55 13 2151 4.99 2.92 0.55 0.79 14 2154 4.92 2.74 0.55 0.78 15 2135 3.91 2.99 0.43 0.69 16 2143 4.81 2.97 0.53 0.80 1 2272 4.67 3.07 0.52 0.67 2 2273 4.12 2.57 0.46 0.66 3 2262 5.11 3.13 0.57 0.83 4 2260 4.03 2.93 0.45 0.75 5 2262 5.46 3.28 0.61 0.80 6 2269 4.15 3.04 0.46 0.67 7 2260 4.95 2.92 0.55 0.78

Reading 5 8 9

2262 2271

4.76 5.78

3.38 3.26

0.53 0.64

0.78 0.84

10 2266 4.64 2.92 0.52 0.69 11 2255 4.75 3.21 0.53 0.77 12 2267 4.73 3.29 0.53 0.81 13 2258 5.48 3.40 0.61 0.83 14 2256 4.76 3.33 0.53 0.78 15 2254 3.97 2.62 0.44 0.66 16 2258 4.36 2.87 0.48 0.71

(cont‘d)


Content

Area Grade Item Position N Mean Rubric

Score s.d. Difficulty Discrimination

1 2195 5.08 3.14 0.56 0.63 2 2183 6.07 3.41 0.67 0.82 3 2187 4.25 2.70 0.47 0.61 4 2194 5.81 2.90 0.65 0.76 5 2178 4.53 2.83 0.50 0.66 6 2187 5.17 3.38 0.57 0.75 7 2177 5.59 3.38 0.62 0.81

Reading 6 8 9

2174 2182

6.30 5.85

3.30 3.10

0.70 0.65

0.85 0.80

10 2174 4.70 2.94 0.52 0.74 11 2166 4.02 2.70 0.45 0.67 12 2171 4.98 3.04 0.55 0.76 13 2173 4.65 2.93 0.52 0.79 14 2165 5.33 3.19 0.59 0.76 15 2155 4.79 3.40 0.53 0.76 16 2158 5.35 3.27 0.59 0.81 1 2368 5.87 3.17 0.65 0.79 2 2366 5.15 3.01 0.57 0.72 3 2362 5.50 3.19 0.61 0.82 4 2357 6.55 3.21 0.73 0.84 5 2362 5.57 3.18 0.62 0.81 6 2359 5.88 3.37 0.65 0.84 7 2347 5.65 3.33 0.63 0.85

Reading 7 8 9

2346 2345

5.46 5.08

3.17 3.19

0.61 0.56

0.82 0.81

10 2350 5.46 3.12 0.61 0.74 11 2338 4.66 3.28 0.52 0.70 12 2347 6.22 2.98 0.69 0.79 13 2357 6.01 3.24 0.67 0.80 14 2360 4.55 2.69 0.51 0.64 15 2343 5.81 3.29 0.65 0.79 16 2335 5.41 3.13 0.60 0.81 1 2379 4.67 2.96 0.52 0.60 2 2374 5.61 3.12 0.62 0.80 3 2367 5.14 3.21 0.57 0.70 4 2376 4.91 3.13 0.55 0.76 5 2377 5.35 3.14 0.59 0.81 6 2376 6.06 3.44 0.67 0.80 7 2381 5.32 2.57 0.59 0.79

Reading 8 8 9

2381 2388

5.46 5.06

3.00 3.18

0.61 0.56

0.78 0.80

10 2369 5.71 3.27 0.63 0.83 11 2387 4.34 2.70 0.48 0.77 12 2365 5.03 3.20 0.56 0.79 13 2376 4.97 3.11 0.55 0.78 14 2370 4.56 2.84 0.51 0.76 15 2375 5.36 2.98 0.60 0.77 16 2361 4.38 3.00 0.49 0.75

(cont‘d)




1 2816 4.60 2.89 0.51 0.65 2 2824 5.58 3.17 0.62 0.79 3 2821 5.94 3.28 0.66 0.78 4 2828 5.90 3.17 0.66 0.77 5 2814 6.23 3.27 0.69 0.75 6 2829 5.91 3.29 0.66 0.77 7 2817 6.52 3.08 0.72 0.84

Reading 9 8 9

2814 2826

5.53 5.08

3.24 3.05

0.61 0.56

0.78 0.70

10 2823 6.12 3.29 0.68 0.83 11 2820 5.44 3.08 0.60 0.79 12 2819 5.43 2.97 0.60 0.80 13 2823 5.94 3.35 0.66 0.83 14 2816 6.24 3.37 0.69 0.86 15 2828 6.08 3.21 0.68 0.82 16 2826 5.17 2.91 0.57 0.75 1 2921 3.57 2.37 0.40 0.61 2 2944 5.96 2.99 0.66 0.79 3 2924 6.03 3.30 0.67 0.80 4 2938 5.76 3.04 0.64 0.79 5 2928 5.72 3.08 0.64 0.82 6 2927 5.91 3.13 0.66 0.79 7 2921 4.99 3.02 0.55 0.74

Reading 10 8 9

2932 2929

5.95 4.42

3.04 2.43

0.66 0.49

0.82 0.58

10 2926 5.98 3.16 0.66 0.77 11 2926 5.93 3.24 0.66 0.80 12 2938 5.38 3.13 0.60 0.78 13 2927 6.10 2.97 0.68 0.82 14 2921 5.82 2.97 0.65 0.78 15 2924 6.07 3.15 0.67 0.82 16 2911 4.86 2.86 0.54 0.69



Grade and Content Area – Reading (Participatory Students) Content Item Mean Rubric Grade N s.d. Difficulty Discrimination Area Position Score

1 852 2.15 1.77 0.24 0.50 2 852 1.86 1.45 0.21 0.59 3 852 2.37 2.11 0.26 0.59 4 852 2.14 1.70 0.24 0.61 5 852 1.91 1.54 0.21 0.54 6 852 1.94 1.66 0.22 0.50 7 852 1.97 1.77 0.22 0.54

Reading 3 8 9

852 852

1.87 1.91

1.43 1.66

0.21 0.21

0.60 0.53

10 852 2.71 2.44 0.30 0.60 11 852 1.83 1.43 0.20 0.55 12 852 2.25 1.79 0.25 0.62 13 852 2.11 1.82 0.23 0.58 14 852 2.01 1.88 0.22 0.55 15 852 1.86 1.54 0.21 0.55 16 852 1.97 1.76 0.22 0.58 1 922 2.67 2.12 0.30 0.59 2 922 2.26 1.94 0.25 0.47 3 922 2.81 2.15 0.31 0.64 4 922 1.88 1.53 0.21 0.47 5 922 1.92 1.59 0.21 0.50 6 922 1.93 1.59 0.21 0.48 7 922 1.86 1.68 0.21 0.43

Reading 4 8 9

922 922

2.33 2.15

1.79 1.87

0.26 0.24

0.54 0.50

10 922 2.07 1.66 0.23 0.49 11 922 2.04 1.67 0.23 0.50 12 922 1.98 1.76 0.22 0.47 13 922 2.36 2.03 0.26 0.59 14 922 2.65 1.97 0.29 0.62 15 922 1.81 1.57 0.20 0.46 16 922 2.32 1.83 0.26 0.60 1 913 2.29 1.84 0.25 0.54 2 913 2.30 1.74 0.26 0.59 3 913 2.28 1.76 0.25 0.60 4 913 1.78 1.24 0.20 0.62 5 913 2.45 2.11 0.27 0.59 6 913 1.88 1.48 0.21 0.45 7 913 2.40 1.99 0.27 0.63

Reading 5 8 9

913 913

1.88 2.47

1.48 1.98

0.21 0.27

0.50 0.62

10 913 2.41 1.81 0.27 0.62 11 913 1.96 1.61 0.22 0.52 12 913 1.78 1.30 0.20 0.54 13 913 2.10 1.70 0.23 0.60 14 913 1.84 1.50 0.20 0.52 15 913 2.09 1.49 0.23 0.64 16 913 2.16 1.55 0.24 0.63

(cont‘d)




1 727 2.42 1.99 0.27 0.55 2 727 2.17 2.02 0.24 0.56 3 727 2.24 1.74 0.25 0.60 4 727 2.78 2.43 0.31 0.64 5 727 2.17 1.69 0.24 0.63 6 727 2.00 1.52 0.22 0.53 7 727 1.99 1.64 0.22 0.60

Reading 6 8 9

727 727

2.44 2.34

2.23 1.99

0.27 0.26

0.62 0.57

10 727 1.76 1.39 0.20 0.55 11 727 1.86 1.58 0.21 0.50 12 727 2.09 1.74 0.23 0.57 13 727 1.83 1.31 0.20 0.64 14 727 2.16 2.12 0.24 0.51 15 727 1.60 1.34 0.18 0.49 16 727 1.86 1.61 0.21 0.59 1 693 2.23 1.83 0.25 0.56 2 693 2.08 1.75 0.23 0.60 3 693 1.84 1.33 0.20 0.64 4 693 2.35 2.12 0.26 0.61 5 693 1.87 1.51 0.21 0.57 6 693 1.83 1.52 0.20 0.49 7 693 1.65 1.44 0.18 0.52

Reading 7 8 9

693 693

1.85 1.56

1.45 1.31

0.21 0.17

0.59 0.53

10 693 2.23 2.03 0.25 0.59 11 693 1.53 1.40 0.17 0.52 12 693 2.61 2.42 0.29 0.68 13 693 2.18 1.84 0.24 0.64 14 693 1.97 1.49 0.22 0.65 15 693 2.03 1.85 0.23 0.61 16 693 1.83 1.67 0.20 0.61 1 796 2.58 2.15 0.29 0.56 2 796 2.25 1.95 0.25 0.56 3 796 2.24 2.09 0.25 0.54 4 796 1.86 1.46 0.21 0.55 5 796 2.05 1.74 0.23 0.55 6 796 2.24 2.16 0.25 0.50 7 796 2.58 2.06 0.29 0.67

Reading 8 8 9

796 796

2.37 2.06

1.96 1.43

0.26 0.23

0.61 0.68

10 796 2.01 1.72 0.22 0.58 11 796 1.88 1.31 0.21 0.61 12 796 1.81 1.53 0.20 0.53 13 796 1.93 1.69 0.21 0.51 14 796 1.99 1.59 0.22 0.57 15 796 2.34 1.98 0.26 0.60 16 796 1.78 1.38 0.20 0.59

(cont‘d)




Reading 9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

790 790 790 790 790 790 790 790 790 790 790 790 790 790 790 790

2.13 2.08 2.15 2.41 2.43 2.25 2.47 2.01 2.08 2.04 2.08 2.04 1.94 1.81 2.22 2.17

1.69 1.61 1.78 1.95 2.11 1.89 2.17 1.62 1.83 1.66 1.80 1.65 1.51 1.46 1.66 1.60

0.24 0.23 0.24 0.27 0.27 0.25 0.27 0.22 0.23 0.23 0.23 0.23 0.22 0.20 0.25 0.24

0.51 0.53 0.53 0.62 0.59 0.58 0.61 0.57 0.55 0.55 0.58 0.61 0.59 0.57 0.62 0.65

Reading 10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

794 794 794 794 794 794 794 794 794 794 794 794 794 794 794 794

1.70 2.41 2.14 2.27 2.14 2.17 2.02 2.21 2.52 2.27 2.08 1.96 2.49 2.33 2.21 2.07

1.10 1.87 1.68 1.59 1.53 1.91 1.46 1.66 1.74 1.88 1.74 1.46 1.95 1.99 1.63 1.55

0.19 0.27 0.24 0.25 0.24 0.24 0.22 0.25 0.28 0.25 0.23 0.22 0.28 0.26 0.25 0.23

0.51 0.57 0.56 0.63 0.64 0.54 0.65 0.59 0.67 0.60 0.59 0.53 0.66 0.63 0.67 0.67



Grade and Content Area – Reading (Supported Students) Content Item Mean Rubric Grade N s.d. Difficulty Discrimination Area Position Score

1 482 5.17 2.30 0.57 -0.06 2 482 4.24 2.11 0.47 -0.03 3 482 5.92 2.25 0.66 0.09 4 482 4.56 2.10 0.51 0.05 5 482 5.04 2.52 0.56 0.06 6 482 4.10 2.62 0.46 0.00 7 482 5.57 2.69 0.62 0.13

Reading 3 8 9

482 482

4.29 5.66

2.07 2.67

0.48 0.63

0.05 0.09

10 482 6.93 2.12 0.77 -0.04 11 482 3.92 2.18 0.44 -0.02 12 482 4.88 2.16 0.54 -0.05 13 482 5.46 2.56 0.61 0.04 14 482 5.88 2.79 0.65 0.04 15 482 4.49 2.62 0.50 -0.04 16 482 5.43 2.09 0.60 0.14 1 705 5.54 2.19 0.62 0.00 2 705 5.13 2.33 0.57 0.03 3 705 6.32 1.95 0.70 0.03 4 705 5.00 2.95 0.56 0.05 5 705 3.95 2.30 0.44 -0.09 6 705 5.42 2.80 0.60 0.11 7 705 4.63 2.59 0.51 0.08

Reading 4 8 9

705 705

5.40 5.59

2.64 2.48

0.60 0.62

0.09 0.07

10 705 4.06 2.25 0.45 -0.07 11 705 4.88 2.88 0.54 0.06 12 705 3.87 2.41 0.43 -0.14 13 705 6.21 1.87 0.69 0.06 14 705 5.70 1.90 0.63 0.12 15 705 4.07 2.39 0.45 0.04 16 705 5.47 2.22 0.61 0.14 1 626 5.17 2.65 0.57 -0.10 2 626 4.41 1.86 0.49 -0.02 3 626 5.31 2.29 0.59 0.19 4 626 3.69 1.86 0.41 0.05 5 626 6.01 2.44 0.67 -0.03 6 626 4.22 2.48 0.47 -0.08 7 626 5.50 2.05 0.61 0.03

Reading 5 8 9

626 626

4.63 6.90

2.85 2.22

0.51 0.77

0.07 0.22

10 626 5.03 2.34 0.56 -0.02 11 626 4.93 2.70 0.55 0.11 12 626 4.82 2.58 0.54 0.14 13 626 6.37 2.61 0.71 0.12 14 626 5.04 2.70 0.56 0.16 15 626 4.12 1.99 0.46 -0.13 16 626 4.53 2.20 0.50 -0.05

(cont‘d)




1 573 5.41 2.61 0.60 -0.09 2 573 6.58 2.68 0.73 0.07 3 573 4.17 1.96 0.46 -0.13 4 573 6.54 1.88 0.73 0.06 5 573 4.51 2.13 0.50 -0.08 6 573 4.48 2.81 0.50 -0.01 7 573 5.55 2.80 0.62 0.16

Reading 6 8 9

573 573

6.92 6.46

2.39 2.17

0.77 0.72

0.24 0.02

10 573 4.89 2.27 0.54 0.03 11 573 3.58 2.11 0.40 -0.09 12 573 4.59 2.22 0.51 0.01 13 573 4.17 1.98 0.46 0.08 14 573 5.33 2.51 0.59 0.01 15 573 3.88 2.66 0.43 0.10 16 573 5.26 2.41 0.58 0.09 1 471 5.55 2.54 0.62 0.07 2 471 4.89 2.26 0.54 -0.12 3 471 4.72 2.15 0.52 0.07 4 471 6.86 2.42 0.76 0.00 5 471 4.91 2.50 0.55 0.09 6 471 4.94 2.81 0.55 0.16 7 471 4.72 2.50 0.52 0.18

Reading 7 8 9

471 471

4.66 3.96

2.39 2.43

0.52 0.44

0.09 0.16

10 471 5.05 2.57 0.56 -0.08 11 471 3.76 2.49 0.42 0.00 12 471 6.77 1.95 0.75 0.03 13 471 5.82 2.49 0.65 0.06 14 471 4.78 2.04 0.53 -0.02 15 471 5.40 2.71 0.60 -0.07 16 471 4.83 2.28 0.54 0.03 1 666 4.55 2.35 0.51 -0.16 2 666 5.97 2.47 0.66 0.10 3 666 5.09 2.62 0.57 -0.10 4 666 4.86 2.60 0.54 0.07 5 666 5.34 2.32 0.59 0.19 6 666 6.46 2.82 0.72 0.12 7 666 5.95 1.58 0.66 0.04

Reading 8 8 9

666 666

5.71 4.25

2.25 2.14

0.63 0.47

0.05 0.08

10 666 6.03 2.43 0.67 0.17 11 666 3.92 1.88 0.44 0.07 12 666 4.83 2.52 0.54 0.05 13 666 4.76 2.39 0.53 0.05 14 666 4.17 2.03 0.46 0.07 15 666 5.56 2.16 0.62 0.00 16 666 3.72 2.03 0.41 0.04

(cont‘d)




Reading 9

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16

583 583 583 583 583 583 583 583 583 583 583 583 583 583 583 583

3.77 4.62 5.50 5.23 6.01 5.15 6.61 4.49 4.45 5.37 4.65 4.91 4.90 5.78 5.40 4.54

1.88 2.38 2.73 2.50 2.93 2.68 2.08 2.27 2.26 2.71 2.10 2.05 2.69 2.82 2.64 2.00

0.42 0.51 0.61 0.58 0.67 0.57 0.73 0.50 0.49 0.60 0.52 0.55 0.54 0.64 0.60 0.50

-0.11 0.12 0.02 0.00 -0.08 -0.01 0.06 -0.01 -0.04 0.17 0.07 0.01 0.11 0.28 0.11 -0.05

Reading 10

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16

773 773 773 773 773 773 773 773 773 773 773 773 773 773 773 773

3.02 5.72 5.37 5.29 4.96 5.65 4.12 5.68 4.19 5.73 5.44 4.53 5.79 5.66 5.66 4.38

1.56 2.27 2.82 2.46 2.30 2.48 1.97 2.26 1.65 2.58 2.48 2.36 2.05 2.20 2.45 2.07

0.34 0.64 0.60 0.59 0.55 0.63 0.46 0.63 0.47 0.64 0.60 0.50 0.64 0.63 0.63 0.49

0.07 0.01 0.09 0.14 0.16 0.05 0.02 0.12 -0.11 0.01 0.07 0.10 0.03 0.01 0.14 0.02



Grade and Content Area – Reading (Independent Students) Content Item Mean Rubric Grade N s.d. Difficulty Discrimination Area Position Score

1 876 7.08 2.45 0.79 0.17 2 876 6.48 2.58 0.72 0.23 3 876 8.19 1.53 0.91 0.24 4 876 7.56 2.13 0.84 0.26 5 876 7.84 1.98 0.87 0.20 6 876 6.73 2.65 0.75 0.20 7 876 8.00 1.99 0.89 0.22

Reading 3 8 9

876 876

7.11 8.40

2.18 1.49

0.79 0.93

0.29 0.19

10 876 7.62 1.61 0.85 -0.02 11 876 5.74 2.70 0.64 0.19 12 876 7.13 2.18 0.79 0.31 13 876 8.38 1.57 0.93 0.17 14 876 8.40 1.47 0.93 0.09 15 876 6.78 2.80 0.75 0.14 16 876 7.70 1.72 0.86 0.29 1 622 7.31 2.08 0.81 0.21 2 622 6.84 2.22 0.76 0.19 3 622 7.62 1.65 0.85 0.20 4 622 8.16 1.92 0.91 0.12 5 622 6.42 2.80 0.71 0.27 6 622 7.69 1.86 0.85 0.16 7 622 7.21 2.38 0.80 0.23

Reading 4 8 9

622 622

7.78 7.65

2.02 1.80

0.86 0.85

0.14 0.14

10 622 6.35 2.89 0.71 0.27 11 622 8.06 2.06 0.90 0.10 12 622 5.47 2.74 0.61 0.11 13 622 7.50 1.62 0.83 0.20 14 622 7.40 1.69 0.82 0.25 15 622 6.83 2.68 0.76 0.16 16 622 7.78 1.66 0.86 0.21 1 821 6.94 2.52 0.77 0.10 2 821 5.91 2.43 0.66 0.21 3 821 8.10 1.69 0.90 0.27 4 821 6.81 2.60 0.76 0.29 5 821 8.38 1.57 0.93 0.15 6 821 6.62 2.75 0.74 0.18 7 821 7.38 1.90 0.82 0.24

Reading 5 8 9

821 821

8.06 8.61

2.04 1.10

0.90 0.96

0.16 0.12

10 821 6.84 2.50 0.76 0.13 11 821 7.72 1.90 0.86 0.10 12 821 7.95 2.01 0.88 0.24 13 821 8.55 1.42 0.95 0.10 14 821 7.80 2.25 0.87 0.20 15 821 5.96 2.50 0.66 0.18 16 821 6.68 2.56 0.74 0.23

(cont‘d)




1 982 6.84 2.77 0.76 0.15 2 982 8.67 1.24 0.96 0.12 3 982 5.79 2.66 0.64 0.26 4 982 7.63 1.65 0.85 0.18 5 982 6.28 2.59 0.70 0.24 6 982 7.92 2.23 0.88 0.16 7 982 8.29 1.78 0.92 0.14

Reading 6 8 9

982 982

8.81 8.09

0.86 1.52

0.98 0.90

0.12 0.14

10 982 6.77 2.24 0.75 0.16 11 982 5.89 2.33 0.65 0.27 12 982 7.35 2.10 0.82 0.25 13 982 7.01 2.18 0.78 0.33 14 982 7.67 1.92 0.85 0.23 15 982 7.67 2.25 0.85 0.20 16 982 7.98 1.90 0.89 0.15 1 1297 7.93 1.90 0.88 0.21 2 1297 6.89 2.38 0.77 0.30 3 1297 7.74 2.07 0.86 0.28 4 1297 8.68 1.02 0.96 0.20 5 1297 7.79 1.82 0.87 0.19 6 1297 8.38 1.45 0.93 0.23 7 1297 8.12 1.64 0.90 0.28

Reading 7 8 9

1297 1297

7.68 7.37

1.92 1.99

0.85 0.82

0.30 0.30

10 1297 7.34 2.16 0.82 0.29 11 1297 6.66 2.76 0.74 0.26 12 1297 7.95 1.52 0.88 0.16 13 1297 8.13 1.88 0.90 0.10 14 1297 5.84 2.41 0.65 0.18 15 1297 7.98 1.91 0.89 0.26 16 1297 7.53 1.94 0.84 0.34 1 1025 6.38 2.79 0.71 0.34 2 1025 7.98 1.49 0.89 0.26 3 1025 7.41 2.35 0.82 0.18 4 1025 7.32 2.16 0.81 0.28 5 1025 7.93 1.71 0.88 0.25 6 1025 8.76 1.08 0.97 0.05 7 1025 7.03 1.46 0.78 0.37

Reading 8 8 9

1025 1025

7.68 7.92

1.80 2.07

0.85 0.88

0.26 0.29

10 1025 8.36 1.44 0.93 0.19 11 1025 6.52 2.12 0.72 0.39 12 1025 7.66 1.96 0.85 0.29 13 1025 7.48 2.01 0.83 0.34 14 1025 6.80 2.17 0.76 0.33 15 1025 7.57 1.88 0.84 0.28 16 1025 6.84 2.46 0.76 0.36

(cont‘d)




Reading 9

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16

1554 1554 1554 1554 1554 1554 1554 1554 1554 1554 1554 1554 1554 1554 1554 1554

6.17 7.72 8.04 7.92 8.24 8.04 8.54 7.72 6.85 8.48 7.44 7.36 8.36 8.66 8.30 6.93

2.67 2.08 2.03 2.05 1.79 2.07 1.20 2.24 2.46 1.44 2.15 2.00 1.69 1.19 1.64 2.30

0.69 0.86 0.89 0.88 0.92 0.89 0.95 0.86 0.76 0.94 0.83 0.82 0.93 0.96 0.92 0.77

0.34 0.31 0.25 0.27 0.13 0.23 0.25 0.34 0.29 0.18 0.36 0.36 0.27 0.19 0.24 0.36

Reading 10

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16

1461 1461 1461 1461 1461 1461 1461 1461 1461 1461 1461 1461 1461 1461 1461 1461

4.88 8.02 8.50 7.90 8.07 8.09 7.08 8.13 5.59 8.12 8.30 7.70 8.23 7.80 8.38 6.62

2.44 1.66 1.52 1.81 1.68 1.62 2.47 1.61 2.41 1.79 1.74 2.03 1.52 1.75 1.55 2.44

0.54 0.89 0.94 0.88 0.90 0.90 0.79 0.90 0.62 0.90 0.92 0.86 0.91 0.87 0.93 0.74

0.33 0.28 0.16 0.22 0.29 0.26 0.35 0.28 0.28 0.20 0.19 0.27 0.29 0.23 0.22 0.21



Grade and Content Area – Science (All Students) Content Item Mean Rubric Grade N s.d. Difficulty Discrimination Area Position Score

1 2230 6.59 3.16 0.73 0.87 2 2229 5.75 3.33 0.64 0.81 3 2212 5.65 3.30 0.63 0.84 4 2212 6.00 3.42 0.67 0.84 5 2219 5.91 2.91 0.66 0.85 6 2203 4.82 3.40 0.54 0.77 7 2219 6.52 3.37 0.72 0.84

Science 5 8 9

2224 2228

5.74 6.11

3.02 3.39

0.64 0.68

0.79 0.87

10 2215 5.92 3.36 0.66 0.84 11 2206 5.51 3.36 0.61 0.77 12 2213 5.91 3.50 0.66 0.85 13 2186 3.01 2.67 0.33 0.52 14 2212 5.92 3.32 0.66 0.86 15 2215 5.24 3.04 0.58 0.81 16 2189 4.21 2.93 0.47 0.63 1 2355 6.83 3.21 0.76 0.85 2 2356 5.79 3.51 0.64 0.80 3 2359 5.94 3.56 0.66 0.76 4 2359 6.40 3.17 0.71 0.89 5 2355 6.39 3.34 0.71 0.84 6 2353 5.39 3.24 0.60 0.79 7 2355 5.48 2.71 0.61 0.78

Science 8 8 9

2346 2358

4.93 6.50

3.43 3.19

0.55 0.72

0.69 0.87

10 2350 5.87 3.08 0.65 0.75 11 2356 6.02 3.17 0.67 0.82 12 2335 5.95 3.31 0.66 0.86 13 2359 5.90 3.24 0.66 0.79 14 2347 6.16 3.42 0.68 0.86 15 2344 5.56 3.11 0.62 0.83 16 2348 5.75 3.33 0.64 0.83 1 2904 6.04 3.42 0.67 0.73 2 2921 6.29 3.03 0.70 0.80 3 2915 5.50 2.97 0.61 0.80 4 2915 6.64 3.11 0.74 0.80 5 2904 6.05 3.15 0.67 0.77 6 2903 5.92 3.44 0.66 0.75 7 2896 5.18 3.29 0.58 0.72

Science 11 8 9

2906 2899

6.08 5.49

2.82 3.37

0.68 0.61

0.80 0.74

10 2901 6.34 3.18 0.70 0.80 11 2912 6.94 2.86 0.77 0.82 12 2908 4.84 2.80 0.54 0.68 13 2911 6.61 2.99 0.73 0.82 14 2893 6.14 3.30 0.68 0.84 15 2898 5.96 3.25 0.66 0.79 16 2901 6.46 3.09 0.72 0.80

(cont‘d)



Grade and Content Area – Science (Participatory Students) Content Item Mean Rubric Grade N s.d. Difficulty Discrimination Area Position Score

1 667 2.49 2.10 0.28 0.66 2 667 1.94 1.68 0.22 0.56 3 667 1.87 1.52 0.21 0.61 4 667 1.87 1.67 0.21 0.53 5 667 2.32 2.02 0.26 0.70 6 667 1.43 1.19 0.16 0.48 7 667 2.27 2.21 0.25 0.57

Science 5 8 9

667 667

2.22 2.00

1.99 1.67

0.25 0.22

0.62 0.61

10 667 1.94 1.69 0.22 0.63 11 667 1.94 1.77 0.22 0.57 12 667 1.72 1.59 0.19 0.53 13 667 1.35 1.11 0.15 0.51 14 667 1.88 1.59 0.21 0.67 15 667 1.96 1.60 0.22 0.60 16 667 1.75 1.52 0.19 0.59 1 627 2.44 2.30 0.27 0.57 2 627 1.73 1.51 0.19 0.52 3 627 1.90 1.86 0.21 0.47 4 627 1.96 1.67 0.22 0.57 5 627 2.04 1.96 0.23 0.52 6 627 1.82 1.38 0.20 0.67 7 627 2.06 1.99 0.23 0.58

Science 8 8 9

627 627

1.65 2.08

1.61 1.99

0.18 0.23

0.50 0.59

10 627 1.93 1.97 0.21 0.48 11 627 2.04 1.72 0.23 0.63 12 627 1.74 1.57 0.19 0.59 13 627 2.01 1.87 0.22 0.51 14 627 1.59 1.47 0.18 0.52 15 627 1.70 1.56 0.19 0.59 16 627 1.86 1.70 0.21 0.62 1 650 1.95 1.70 0.22 0.56 2 650 2.10 1.82 0.23 0.55 3 650 1.82 1.53 0.20 0.54 4 650 2.23 2.13 0.25 0.57 5 650 2.04 1.96 0.23 0.56 6 650 1.70 1.58 0.19 0.47 7 650 1.70 1.53 0.19 0.49

Science 11 8 9

650 650

2.22 1.71

2.12 1.47

0.25 0.19

0.61 0.56

10 650 2.10 1.93 0.23 0.60 11 650 2.67 2.45 0.30 0.69 12 650 1.83 1.55 0.20 0.61 13 650 2.33 2.16 0.26 0.65 14 650 1.62 1.39 0.18 0.57 15 650 1.76 1.74 0.20 0.47 16 650 1.97 1.83 0.22 0.63

(cont‘d)



Grade and Content Area – Science (Supported Students) Content Item Mean Rubric Grade N s.d. Difficulty Discrimination Area Position Score

1 437 6.81 2.19 0.76 0.16 2 437 5.13 2.53 0.57 -0.01 3 437 4.61 2.40 0.51 0.13 4 437 5.38 2.74 0.60 0.09 5 437 6.10 1.58 0.68 0.26 6 437 3.40 2.34 0.38 0.02 7 437 6.54 2.79 0.73 0.10

Science 5 8 9

437 437

5.76 5.19

2.32 2.70

0.64 0.58

-0.04 0.15

10 437 5.23 2.62 0.58 0.06 11 437 4.99 2.66 0.55 0.01 12 437 5.08 2.86 0.56 0.18 13 437 2.47 1.78 0.27 -0.15 14 437 5.24 2.27 0.58 0.02 15 437 4.42 1.98 0.49 0.16 16 437 3.86 2.04 0.43 -0.02 1 415 6.48 2.70 0.72 0.14 2 415 4.17 2.74 0.46 0.04 3 415 4.62 3.12 0.51 -0.12 4 415 5.93 2.23 0.66 0.16 5 415 5.35 2.84 0.59 0.04 6 415 3.87 1.95 0.43 -0.04 7 415 5.54 2.08 0.62 -0.04

Science 8 8 9

415 415

3.53 6.15

2.68 2.31

0.39 0.68

-0.14 0.12

10 415 6.23 2.63 0.69 -0.04 11 415 5.36 2.13 0.60 0.12 12 415 4.55 2.46 0.51 0.16 13 415 5.08 2.81 0.56 -0.01 14 415 5.22 2.84 0.58 0.20 15 415 4.72 2.12 0.52 0.18 16 415 4.19 2.23 0.47 0.01 1 598 4.80 2.93 0.53 -0.02 2 598 5.64 2.46 0.63 -0.07 3 598 4.21 2.01 0.47 0.13 4 598 6.19 2.53 0.69 0.13 5 598 5.13 2.71 0.57 0.08 6 598 4.44 2.94 0.49 0.00 7 598 3.36 2.15 0.37 0.00

Science 11 8 9

598 598

5.70 3.85

1.99 2.47

0.63 0.43

0.16 -0.03

10 598 5.36 2.72 0.60 0.06 11 598 6.92 1.90 0.77 0.05 12 598 4.08 1.93 0.45 -0.07 13 598 6.23 2.22 0.69 0.09 14 598 4.71 2.58 0.52 0.13 15 598 4.70 2.72 0.52 0.07 16 598 6.11 2.47 0.68 0.04

(cont‘d)



Grade and Content Area – Science (Independent Students) Content Item Mean Rubric Grade N s.d. Difficulty Discrimination Area Position Score

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15

Science 5

16 1 2 3 4 5 6 7 8 9

10 11 12 13 14 15

Science 8

16 1 2 3 4 5 6 7 8 9

10 11 12 13 14 15

Science 11

16

1209 1209 1209 1209 1209 1209 1209 1209 1209 1209 1209 1209 1209 1209 1209 1209 1412 1412 1412 1412 1412 1412 1412 1412 1412 1412 1412 1412 1412 1412 1412 1412 1769 1769 1769 1769 1769 1769 1769 1769 1769 1769 1769 1769 1769 1769 1769 1769

8.78 8.09 8.11 8.51 7.82 7.21 8.86 7.68 8.72 8.36 7.66 8.52 4.12 8.40 7.35 5.70 8.89 8.07 8.13 8.52 8.63 7.42 6.98 6.79 8.57 7.51 7.99 8.23 7.86 8.47 7.53 7.94 7.96 8.06 7.29 8.42 7.84 7.97 7.08 7.62 7.43 8.23 8.51 6.20 8.32 8.29 7.94 8.23

0.92 0.98 0.19 1.92 0.90 0.25 1.78 0.90 0.32 1.50 0.95 0.22 1.50 0.87 0.24 2.57 0.80 0.36 0.84 0.98 0.08 1.64 0.85 0.17 1.15 0.97 0.23 1.63 0.93 0.24 2.35 0.85 0.32 1.51 0.95 0.27 3.00 0.46 0.42 1.57 0.93 0.27 2.07 0.82 0.33 2.85 0.63 0.28 0.73 0.99 0.16 2.21 0.90 0.30 2.27 0.90 0.21 1.18 0.95 0.37 1.25 0.96 0.21 2.42 0.82 0.42 1.52 0.78 0.29 2.89 0.75 0.41 1.19 0.95 0.29 1.76 0.83 0.14 1.94 0.89 0.28 1.59 0.91 0.33 1.89 0.87 0.29 1.45 0.94 0.29 1.91 0.84 0.32 2.06 0.88 0.41 2.38 0.88 0.23 1.64 0.90 0.30 2.03 0.81 0.45 1.52 0.94 0.20 1.91 0.87 0.25 2.22 0.89 0.20 2.63 0.79 0.38 1.64 0.85 0.33 2.56 0.83 0.36 1.71 0.91 0.29 1.15 0.95 0.23 2.41 0.69 0.34 1.46 0.92 0.26 1.67 0.92 0.31 1.85 0.88 0.30 1.56 0.91 0.19

(cont‘d)



Grade and Content Area – Writing (All Students) Content Item Mean Rubric Grade N s.d. Difficulty Discrimination Area Position Score

1 2139 5.46 3.16 0.61 0.82 2 2142 4.01 2.32 0.45 0.65 3 2138 4.61 2.67 0.51 0.75 4 2140 4.95 3.08 0.55 0.82 5 2132 5.09 3.09 0.57 0.83 6 2131 3.98 2.52 0.44 0.76 7 2144 4.38 2.65 0.49 0.77

Writing 4 8 9

2112 2128

4.63 5.11

3.38 3.17

0.51 0.57

0.78 0.85

10 2139 4.63 2.82 0.51 0.79 11 2138 5.09 3.04 0.57 0.84 12 2128 4.43 2.81 0.49 0.78 13 2130 5.14 2.84 0.57 0.84 14 2129 5.54 3.17 0.62 0.82 15 2132 5.02 2.99 0.56 0.83 16 2131 4.53 2.93 0.50 0.82 1 2320 4.52 2.94 0.50 0.73 2 2318 5.52 2.98 0.61 0.82 3 2308 5.52 3.22 0.61 0.83 4 2314 4.98 2.79 0.55 0.83 5 2310 5.25 3.05 0.58 0.87 6 2321 5.04 2.84 0.56 0.81 7 2314 5.20 3.01 0.58 0.84

Writing 8 8 9

2310 2309

5.13 4.77

2.93 2.90

0.57 0.53

0.85 0.83

10 2311 5.36 3.21 0.60 0.84 11 2311 4.12 2.79 0.46 0.77 12 2305 4.92 2.92 0.55 0.84 13 2310 5.50 3.06 0.61 0.85 14 2297 4.30 3.14 0.48 0.77 15 2303 5.85 3.10 0.65 0.85 16 2300 4.86 3.10 0.54 0.86 1 2902 5.47 2.83 0.61 0.83 2 2900 4.97 2.73 0.55 0.65 3 2898 6.79 2.95 0.75 0.80 4 2893 4.92 2.74 0.55 0.76 5 2884 5.73 3.42 0.64 0.80 6 2870 5.16 3.03 0.57 0.83 7 2878 4.61 2.94 0.51 0.73

Writing 10 8 9

2917 2887

5.36 4.37

2.86 2.92

0.60 0.49

0.81 0.75

10 2895 5.20 2.76 0.58 0.76 11 2892 4.75 2.67 0.53 0.67 12 2893 5.51 2.93 0.61 0.81 13 2869 4.89 3.14 0.54 0.79 14 2888 4.17 2.40 0.46 0.61 15 2872 5.44 2.88 0.60 0.82 16 2867 4.20 2.63 0.47 0.76

(cont‘d)



Grade and Content Area – Writing (Participatory Students) Content Item Mean Rubric Grade N s.d. Difficulty Discrimination Area Position Score

1 856 2.46 1.93 0.27 0.63 2 856 2.42 1.58 0.27 0.66 3 856 2.43 1.76 0.27 0.65 4 856 2.14 1.48 0.24 0.65 5 856 2.20 1.46 0.24 0.66 6 856 2.02 1.30 0.22 0.68 7 856 2.14 1.40 0.24 0.68

Writing 4 8 9

856 856

1.76 2.06

1.35 1.52

0.20 0.23

0.53 0.64

10 856 2.24 1.60 0.25 0.65 11 856 2.16 1.48 0.24 0.67 12 856 2.20 1.36 0.24 0.72 13 856 2.40 1.58 0.27 0.71 14 856 2.56 2.02 0.28 0.66 15 856 2.34 1.57 0.26 0.70 16 856 2.04 1.24 0.23 0.71 1 870 2.11 1.57 0.23 0.56 2 870 2.59 1.94 0.29 0.71 3 870 2.20 1.76 0.24 0.63 4 870 2.34 1.77 0.26 0.62 5 870 2.19 1.67 0.24 0.66 6 870 2.43 1.78 0.27 0.67 7 870 2.23 1.56 0.25 0.69

Writing 8 8 9

870 870

2.17 1.97

1.59 1.39

0.24 0.22

0.66 0.63

10 870 2.15 1.74 0.24 0.58 11 870 1.80 1.34 0.20 0.56 12 870 2.12 1.38 0.24 0.72 13 870 2.36 1.55 0.26 0.68 14 870 1.73 1.20 0.19 0.60 15 870 2.63 2.17 0.29 0.69 16 870 1.94 1.25 0.22 0.69 1 990 2.54 1.64 0.28 0.67 2 990 2.93 1.85 0.33 0.66 3 990 3.51 2.41 0.39 0.72 4 990 2.56 1.59 0.28 0.68 5 990 2.00 1.52 0.22 0.53 6 990 2.10 1.34 0.23 0.67 7 990 2.00 1.51 0.22 0.51

Writing 10 8 9

990 990

2.54 1.96

1.27 1.15

0.28 0.22

0.70 0.66

10 990 2.47 1.69 0.27 0.64 11 990 2.63 1.77 0.29 0.62 12 990 2.47 1.63 0.27 0.68 13 990 1.91 1.20 0.21 0.63 14 990 2.57 1.64 0.29 0.64 15 990 2.43 1.71 0.27 0.66 16 990 2.01 1.16 0.22 0.72

(cont‘d)



Grade and Content Area – Writing (Supported Students) Content Item Mean Rubric Grade N s.d. Difficulty Discrimination Area Position Score

1 602 6.11 2.33 0.68 0.08 2 602 4.21 1.66 0.47 -0.08 3 602 5.01 1.90 0.56 0.02 4 602 5.25 2.22 0.58 0.13 5 602 5.41 2.39 0.60 0.20 6 602 3.94 1.63 0.44 0.10 7 602 4.65 1.89 0.52 0.05

Writing 4 8 9

602 602

4.40 5.52

2.90 2.25

0.49 0.61

0.07 0.12

10 602 4.79 1.82 0.53 0.10 11 602 5.62 2.16 0.62 0.15 12 602 4.22 1.86 0.47 0.05 13 602 5.79 1.83 0.64 0.08 14 602 6.22 2.33 0.69 0.09 15 602 5.15 2.00 0.57 0.06 16 602 4.22 1.91 0.47 0.12 1 563 4.28 1.96 0.48 -0.04 2 563 5.91 2.00 0.66 0.02 3 563 5.98 2.40 0.66 0.09 4 563 4.96 1.74 0.55 0.13 5 563 5.15 1.84 0.57 0.12 6 563 5.10 1.95 0.57 0.13 7 563 5.27 2.14 0.59 0.16

Writing 8 8 9

563 563

5.35 4.73

1.93 2.01

0.59 0.53

0.17 0.15

10 563 5.38 2.32 0.60 0.06 11 563 3.72 1.94 0.41 0.09 12 563 4.79 1.96 0.53 0.22 13 563 5.76 2.22 0.64 0.22 14 563 3.47 2.10 0.39 0.06 15 563 6.40 1.85 0.71 0.05 16 563 4.18 1.99 0.46 0.22 1 851 5.59 1.86 0.62 0.20 2 851 4.92 2.06 0.55 -0.11 3 851 7.83 1.91 0.87 0.13 4 851 4.65 1.83 0.52 0.10 5 851 6.22 2.88 0.69 0.13 6 851 5.00 2.17 0.56 0.23 7 851 4.48 2.38 0.50 0.07

Writing 10 8 9

851 851

5.23 3.67

2.19 1.74

0.58 0.41

0.18 0.07

10 851 5.59 2.16 0.62 0.04 11 851 4.76 2.12 0.53 0.00 12 851 5.80 2.02 0.64 0.03 13 851 4.55 2.43 0.51 0.18 14 851 4.00 1.71 0.44 -0.06 15 851 5.67 1.91 0.63 0.10 16 851 3.66 1.49 0.41 0.14

(cont‘d)



Grade and Content Area – Writing (Independent Students) Content Item Mean Rubric Grade N s.d. Difficulty Discrimination Area Position Score

Writing 4

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16

761 761 761 761 761 761 761 761 761 761 761 761 761 761 761 761

8.33 5.64 6.76 7.87 8.08 6.22 6.68 8.03 8.22 7.19 7.98 7.11 7.72 8.35 7.95 7.56

1.41 2.28 2.07 1.95 1.58 2.30 2.10 1.94 1.57 2.17 1.63 2.34 1.63 1.45 1.85 2.03

0.93 0.63 0.75 0.87 0.90 0.69 0.74 0.89 0.91 0.80 0.89 0.79 0.86 0.93 0.88 0.84

0.22 0.21 0.36 0.21 0.17 0.41 0.25 0.22 0.27 0.30 0.24 0.32 0.30 0.23 0.32 0.30

Writing 8

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16

978 978 978 978 978 978 978 978 978 978 978 978 978 978 978 978

6.79 7.89 8.20 7.34 8.02 7.33 7.81 7.64 7.28 8.20 6.41 7.49 8.15 7.06 8.40 7.86

2.56 1.71 1.53 1.73 1.59 1.90 1.68 1.65 1.86 1.56 2.29 1.88 1.54 2.57 1.28 1.76

0.75 0.88 0.91 0.82 0.89 0.81 0.87 0.85 0.81 0.91 0.71 0.83 0.91 0.78 0.93 0.87

0.31 0.26 0.19 0.43 0.38 0.34 0.31 0.34 0.37 0.26 0.40 0.33 0.17 0.35 0.32 0.36

Writing 10

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16

1144 1144 1144 1144 1144 1144 1144 1144 1144 1144 1144 1144 1144 1144 1144 1144

7.92 6.76 8.85 7.17 8.61 7.92 6.96 7.91 6.97 7.28 6.56 7.93 7.73 5.67 7.87 6.48

1.61 2.55 0.78 2.21 1.36 1.81 2.23 1.73 2.60 1.76 2.31 1.76 2.07 2.45 1.63 2.38

0.88 0.75 0.98 0.80 0.96 0.88 0.77 0.88 0.77 0.81 0.73 0.88 0.86 0.63 0.87 0.72

0.26 0.30 0.04 0.30 0.08 0.24 0.27 0.27 0.33 0.24 0.23 0.25 0.20 0.31 0.23 0.37


APPENDIX J—DECISION ACCURACY AND CONSISTENCY

Appendix J—Decision Accuracy & Consistency 227 2007–08 Florida Alt Technical Report

Table J-1a. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 3 Mathematics

Accuracy and Consistency of Classification Indices Accuracy Consistency Kappa (κ)Overall Indices 0.7877 0.7088 0.5650

Accuracy Consistency below P-P 0.7237 0.6044Indices Conditional on P-P thru S-B 0.8023 0.7541Level S-P thru I-B 0.7613 0.6846

I-P and above 0.9362 0.8572

Accuracy Consistency Indices for Accuracy False Positives False Negatives

Dichotomous Decisions P-B : P-P 0.9550 0.0211 0.0239 0.9374 Around Cut Points S-B : S-P 0.8967 0.0637 0.0396 0.8576

I-B : I-P 0.9360 0.0489 0.0151 0.9126 LOCs: P- = Participatory; S- = Supported; I- = Independent. Performance Levels: -B=Basic, -P = Proficient.

Table J-1b 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 4 Mathematics


Indices Conditional on Level

Indices for Dichotomous Decisions

Around Cut Points

below P-P P-P thru S-B S-P thru I-B

I-P and above

Accuracy P-B : P-P 0.9591 S-B : S-P 0.9040 I-B : I-P 0.9302

Accuracy Consistency 0.7378 0.6243 0.8025 0.7512 0.7612 0.6872 0.8764 0.7372

Accuracy Consistency False Positives False Negatives

0.0193 0.0216 0.9429 0.0580 0.0380 0.8673 0.0515 0.0184 0.9045

LOCs: P- = Participatory; S- = Supported; I- = Independent. Performance Levels: -B=Basic, -P = Proficient.

Table J-1c. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 5 Mathematics




Around Cut Points


I-P and above




0.0350 0.0308 0.9103 0.0572 0.0345 0.8734 0.0491 0.0179 0.9087



Table J-1d. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 6 Mathematics



I-P and above 0.8696 0.7432




Table J-1e. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 7 Mathematics




Around Cut Points


I-P and above




0.0230 0.0248 0.9335 0.0593 0.0393 0.8642 0.0586 0.0208 0.8921


Table J-1f. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 8 Mathematics




Around Cut Points


I-P and above




0.0232 0.0256 0.9318 0.0602 0.0404 0.8612 0.0531 0.0187 0.9019



Table J-1g. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 9 Mathematics



I-P and above 0.8964 0.7847




Table J-1h. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 10 Mathematics




Around Cut Points


I-P and above




0.0208 0.0238 0.9377 0.0558 0.0406 0.8670 0.0619 0.0243 0.8826



Table J-2a. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 3 Reading



I-P and above 0.8896 0.8013




Table J-2b. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 4 Reading




Around Cut Points


I-P and above




0.0278 0.0285 0.9216 0.0629 0.0391 0.8595 0.0564 0.0207 0.8947


Table J-2c. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 5 Reading




Around Cut Points


I-P and above




0.0334 0.0271 0.9164 0.0532 0.0342 0.8792 0.0631 0.0301 0.8734



Table J-2d. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 6 Reading



I-P and above 0.8693 0.7707




Table J-2e. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 7 Reading




Around Cut Points


I-P and above




0.0307 0.0210 0.9293 0.0421 0.0274 0.9043 0.0671 0.0434 0.8483


Table J-2f. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 8 Reading




Around Cut Points


I-P and above




0.0235 0.0233 0.9350 0.0548 0.0363 0.8746 0.0647 0.0251 0.8789



Table J-2g. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 9 Reading



I-P and above 0.9160 0.8405




Table J-2h. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 10 Reading




Around Cut Points


I-P and above




0.0178 0.0191 0.9486 0.0496 0.0349 0.8836 0.0687 0.0279 0.8702



Table J-3a. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 5 Science



I-P and above 0.9124 0.8467




Table J-3b. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 8 Science




Around Cut Points


I-P and above




0.0302 0.0189 0.9330 0.0379 0.0238 0.9152 0.0592 0.0327 0.8754


Table J-3c. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 11 Science




Around Cut Points


I-P and above




0.0166 0.0170 0.9536 0.0442 0.0313 0.8964 0.0720 0.0301 0.8643



Table J-3d. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 4 Writing Accuracy and Consistency of Classification Indices

Accuracy Consistency Kappa (κ)Overall Indices 0.8040 0.7305 0.6200


I-P and above 0.8975 0.7944




Table J-3e 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 8 Writing Accuracy and Consistency of Classification Indices

Accuracy Consistency Kappa (κ)Overall Indices 0.7939 0.7190 0.6096



Around Cut Points


I-P and above




0.0248 0.0220 0.9352 0.0470 0.0315 0.8915 0.0556 0.0253 0.8901


Table J-3f. 2007–08 Florida Alternate Assessment: Accuracy and Consistency—Grade 10 Writing




Around Cut Points


I-P and above




0.0162 0.0187 0.9511 0.0524 0.0368 0.8765 0.0581 0.0248 0.8869




Florida Alternate Assessment Technical Report 2007 … · Florida Alternate Assessment Technical Report 2007-2008 . Prepared by Measured Progress for the Florida Department of Education

Documents