Standard Setting Report JUNE 28–30, 2010 PORTLAND, ME Prepared for the Maine Department of Education by: 100 Education Way, Dover, NH 03820 (800) 431-8901
Standard Setting Report
JUNE 28–30, 2010
PORTLAND, ME
Prepared for the Maine Department of Education by:
100 Education Way, Dover, NH 03820 (800) 431-8901
TABLE OF CONTENTS
CHAPTER 1. OVERVIEW OF STANDARD SETTING ..........................................................................................................2
CHAPTER 2. TASKS COMPLETED PRIOR TO STANDARD SETTING ..................................................................................3
2.1 Creation of Achievement Level Descriptors .................................................................................................3
2.2 Selection of Student Portfolios (Bodies of Work)..........................................................................................3
2.3 Preparation of Materials for Panelists .........................................................................................................3
2.4 Preparation of Presentation Materials .........................................................................................................4
2.5 Preparation of Instructions for Facilitators Document ................................................................................4 2.6 Preparation of Systems and Materials for Analysis During the Meeting .....................................................4
2.7 Selection of Panelists .............................................................................................................. ......................4 CHAPTER 3. TASKS COMPLETED DURING STANDARD SETTING ....................................................................................6
3.2 Orientation................................................................................................................... .................................6
3.3 Review of Assessment Materials ...................................................................................................................6 3.4 Review of Achievement Level Definitions .....................................................................................................7
3.5 Training Evaluation ............................................................................................................................. .........7
3.6 Round 1 Judgments .......................................................................................................................................7 3.7 Tabulation of Round 1 Results ......................................................................................................................7
3.8 Round 2 Judgments ............................................................................................................................. ........10
3.9 Tabulation of Round 2 Results ....................................................................................................................10 3.10 Round 3 Judgments .................................................................................................................. ...................13
3.11 Tabulation of Round 3 Results ....................................................................................................................13
3.12 Repeat Process for Second Grade and Content Area (Mathematics and Reading Only) ........................... 21 3.13 Evaluation............................................................................................................................. ......................21
CHAPTER 4. TASKS COMPLETED AFTER STANDARD SETTING ....................................................................................22
4.1 Analysis and Review of Panelists’ Feedback ..............................................................................................22
4.2 Calculation of Recommended Cutpoints.....................................................................................................22
4.3 Preparation of this Standard Setting Report...............................................................................................27
REFERENCES .................................................................................................................................................................28
APPENDICES...................................................................................................................................................................29
APPENDIX A. ACHIEVEMENT LEVEL DEFINITIONS
APPENDIX B. AGENDAS
APPENDIX C. NONDISCLOSURE FORM
APPENDIX D. SAMPLE RATING FORM
APPENDIX E. SAMPLE EVALUATION
APPENDIX F. OPENING SESSION POWERPOINT
APPENDIX G. FACILITATORS’ SCRIPTS
APPENDIX H. PANELIST AFFILIATIONS
APPENDIX I. EVALUATION RESULTS
APPENDIX J. POLICY ADJUSTMENTS
Overview of Standard Setting Process 1 2010 Maine PAAP Standard Setting Report
Chapter 1. OVERVIEW OF STANDARD SETTING
The purpose of this report is to summarize the activities of the standard setting meeting for Maine’s
Personalized Alternate Assessment Portfolio (PAAP) in reading and mathematics (grades 2–7, 10, and 11),
science (grades 5, 8, and 11) and writing (grades 4, 7, and 11). The PAAP standard setting meeting was held
between June 27 and 29, 2010. In all, there were 14 panels with 70 panelists participating in the process.
Eight panels met for two days, and each panel established cuts for two grade level combinations (either two
reading grades or two mathematics grades). The remaining six panels met for one day and established cuts for
a single grade and content area combination. The configuration of the panels is shown in Table 1-1. Note that
some panelists participated in multiple content areas.
Table 1-1. 2010 PAAP Standard Setting: Configuration of Standard Setting Panels
Panel Num
pan
ber of
elists June 28
June 29
June 30
Panel 1 8 Math 2 Math 3 Panel 2 8 Math 4 Math 5 Panel 3 1 0 Math 6 Math 7 Panel 4 9 Math 10 Math 11 Panel 5 1 1 Reading 2 Reading 3 Panel 6 9 Reading 4 Reading 5 Panel 7 8 Reading 6 Reading 7 Panel 8 7 Reading 10 Reading 11 Panel 9 7 Science 5 Panel 10 7 Science 8 Panel 11 7 Science 11 Panel 12 1 0 Writing 4 Panel 13 9 Writing 7 Panel 14 8 Writing 11
A modified version of the body of work method was used for setting standards for the PAAP. The
body of work standard setting method was developed specifically for use with assessments that are designed to
allow for a range of student responses, such as portfolio- or performance-based assessments (Kingston, Kahl,
Sweeney, & Bay, 2001). A modified version of the method has been in use for a number of years that
substantially reduces the logistical burden of the procedure and has been found to yield reasonable and
defensible cutpoints. In the body of work method, panelists are presented with samples of actual student work
(in this case, student portfolios) and make their judgments based on those samples. Specifically, panelists
examine each student portfolio and determine which achievement level best matches the particular
knowledge, skills, and abilities the student exhibits through his or her performance on the work sample. This
report is organized into three major sections, describing tasks completed prior to, during, and after the
standard setting meeting.
Overview of Standard Setting Process 2 2010 Maine PAAP Standard Setting Report
Chapter 2. TASKS COMPLETED PRIOR TO STANDARD SETTING
2.1 Creation of Achievement Level Descriptors
The Achievement Level Descriptors (ALDs) describe the set of knowledge, skills, and abilities that
students in each achievement level are expected to display. Staff at the Maine Department of Education
(MDOE) created these draft descriptors prior to the standard setting meeting, where they were presented to
the panelists. The draft ALDs are provided in Appendix A.
2.2 Selection of Student Portfolios (Bodies of Work)
The goal in selecting student portfolios to use for the standard setting was to select a total of 30 to 40
bodies of work, spread as evenly as possible across the range of possible total raw scores. For the PAAP,
teachers select from a series of standardized tasks. For future administrations, the teachers will be required to
sample tasks that match a predefined blueprint. Consequently, an additional goal was to sample portfolios that
matched the blueprint as best as possible. However, because the blueprint was not implemented in 2010, it
was not possible to find portfolios that represented the blueprint at each total score. A list of portfolios was
generated with approximately three times the target number of portfolios at each score point, and the
portfolios were reviewed and selected by Measured Progress special education staff. The final numbers of
portfolios selected ranged from 29 to 38.
2.3 Preparation of Materials for Panelists
The following materials were assembled for presentation to the panelists at the standard setting
meeting:
▪ Meeting agenda
▪ Nondisclosure agreement
▪ ALDs
▪ Samples of student portfolios
▪ Rating forms
▪ Evaluation forms
Copies of the ALDs, meeting agenda, nondisclosure form, a sample rating form, and evaluation forms
are included in Appendices A through E.
Tasks Prior to Standard Setting 3 2010 Maine PAAP Standard Setting Report
2.4 Preparation of Presentation Materials
The PowerPoint presentation used in the opening session was prepared prior to the meeting. The
presentation was designed to give panelists an overview of the assessment and how it is scored as well as a
preview of what to expect throughout the standard setting process. A copy of the presentation is included in
Appendix F.
2.5 Preparation of Instructions for Facilitators Document
A script was created for the group facilitators to refer to while working through each step of the
standard setting process. This document is included in Appendix G. The facilitators also attended a training
session, led by a Measured Progress psychometrician, approximately one week before the standard setting
meeting. The purpose of the training was to prepare the facilitators for the panel activities and to ensure
consistency in the implementation of procedures.
2.6 Preparation of Systems and Materials for Analysis During the
Meeting
The computational programming used to carry out all analyses during the standard setting meeting
was completed and thoroughly tested prior to the standard setting meeting.
2.7 Selection of Panelists
As was emphasized in Cizek and Bunch (2007), regardless of the method used, the selection of
panelists is an important factor in determining standard setting outcomes and maximizing the validity of the
standard setting process. The guidance provided by Standards for Educational and Psychological Testing
(1999) states “a sufficiently large and representative group of judges should be involved to provide reasonable
assurance that results would not vary greatly if the process were repeated.” Consistent with the above guidance,
as well as practical considerations regarding the maximum size of group that can be successfully managed by
group facilitators, the goal was to recruit standard setting panels of eight members representing different
stakeholder groups to set standards for each grade and content area. Targets for the size and composition of the
panels were also consistent with federal guidelines as described in Standards and Assessment Peer Review
Guidance: Information and Examples for Meeting Requirements of the No Child Left Behind Act of 2001 (U.S.
Department of Education, 2009).
Panelists were selected by the MDOE and Measured Progress prior to the standard setting meeting.
The goal was for each panel to consist of approximately eight participants: two or three special education
teachers experienced in working with students who have significant disabilities; two or three content area
teachers (representative of a range of grade level experiences); and one or two school administrators, higher
education personnel, general education teachers, or stakeholders from interest groups related to significant
Tasks Prior to Standard Setting 4 2010 Maine PAAP Standard Setting Report
disabilities. In addition, to the extent possible, panels were assembled so as to reflect a balance of gender,
race/ethnicity, and geographic location. A list of the panelists and their affiliations is included in Appendix H.
Tasks Prior to Standard Setting 5 2010 Maine PAAP Standard Setting Report
Chapter 3. TASKS COMPLETED DURING STANDARD SETTING
3.1 Overview of Body of Work Method
The body of work standard setting method was developed specifically for use with assessments that
are designed to allow for a range of student responses, such as portfolio- and performance-based assessments.
For a number of years a modified version of the method has been in use that substantially reduces the logistical
burden of the procedure and has been found to yield reasonable and defensible cutpoints. Panelists were asked
to evaluate each work sample from a holistic perspective before classifying it into a single achievement level.
3.2 Orientation
With regard to panelist training, Standards for Educational and Psychological Testing states the
following:
Care must be taken to assure that judges understand what they are to do. The
process must be such that well-qualified judges can apply their knowledge
and experience to reach meaningful and relevant judgments that accurately
reflect their understanding and intentions. (AERA/APA/NCME, 1999, p. 54)
The training of the panelists began with a general orientation at the start of the standard setting
meeting. The purpose of the orientation was to ensure that all panelists received the same information about
the need for and goals of standard setting and about their part in the process. First, the MDOE provided some
pertinent context about the PAAP program and an introduction to the issues of standard setting. Second, the
PAAP program manager provided an overview of the assessment, including alternate grade level expectations,
task bank selection, administration, and scoring. Next, a Measured Progress psychometrician presented a brief
overview of the body of work procedure and the activities that would occur during the standard setting
meeting. Once the general orientation was complete, each panel convened in a breakout room, where the
panelists received more detailed training and completed the standard setting activities.
3.3 Review of Assessment Materials
The first step after the opening session was for the panelists to become familiar with the PAAP. The
purpose of this step was to make sure the panelists thoroughly understood how the portfolio is administered
and scored. Panelists reviewed the Alternate Grade Level Indicators (AGLIs) and administration manuals. In
addition, panelists individually reviewed every fifth portfolio and discussed the knowledge, skills, and
abilities associated with each work sample as a group.
Tasks During Standard Setting 6 2010 Maine PAAP Standard Setting Report
3.4 Review of Achievement Level Definitions
The second step in the process, once the panelists convened into their content area and grade level
groups, was to discuss the Achievement Level Descriptors (ALDs). This important step was designed to ensure
that panelists thoroughly understood the knowledge, skills, and abilities needed for students to be classified
into achievement levels (Substantially Below Proficient, Partially Proficient, Proficient, and Proficient
With Distinction). Panelists first reviewed the ALDs on their own and then participated in group discussion
of the ALDs, clarifying the description for each achievement level. The discussions focused on the knowledge,
skills, and abilities that differentiated adjacent achievement levels. Bulleted lists of characteristics for each
level were generated based on the group discussion and were posted in the room for panelists to refer to during
the rounds of ratings.
3.5 Training Evaluation
Prior to beginning the Round 1 ratings, the panelists anonymously completed a training evaluation
form. The purpose of the evaluation was to ensure that panelists were comfortable with the process and ready
to move on to the rating task. Any issues or problems that came up in the training evaluations were addressed
before the facilitator proceeded to Round 1.
3.6 Round 1 Judgments
In the first round, panelists worked individually with the ALDs, the student portfolios, and the rating
form. The work samples consisted of 29 to 38 portfolios, with scores covering the full range of possible total
scores. For each portfolio, the panelists considered the skills and abilities demonstrated in the work sample,
and panelists decided which achievement level was the best match. The panelists worked their way through
the portfolios, making a rating for each one, and recorded their ratings on the rating form. While the portfolios
were presented in order of total score, panelists were not required to rate them strictly in increasing order.
Instead, panelists were encouraged to take a holistic look at the knowledge, skills, and abilities demonstrated
in the portfolio, rather than making a judgment based primarily on the total raw score.
3.7 Tabulation of Round 1 Results
After all panelists had completed their individual ratings, the Measured Progress data analysis team
calculated the average cutpoints for the group based on the Round 1 ratings. Cutpoints were calculated using
SAS statistical software. Logistic regression was used to determine each panelist’s individual cutpoints, and
then the cutpoints were averaged across the group. In addition, impact data were calculated, which reflected
the percentage of students who would fall into each achievement level based on the group average Round 1
ratings. The Round 1 results are outlined in Tables 3-1 through 3-4.
Tasks During Standard Setting 7 2010 Maine PAAP Standard Setting Report
Achievement level Average
cut Standard
error Raw
Min
score
Max
Percent of students
PD 62.6 1.6 63 69 3.4 Proficient 45.9 2.1 46 62 21.0 Partially Proficient 24.9 2.5 25 45 50.0
PD 70.2 1.6 NA NA NA Proficient 52.9 2.8 53 69 35.6 Partially Proficient 25.8 1.9 26 52 41.5
PD 65.4 1.1 66 69 12.0 Proficient 42.8 2.3 43 65 50.9 Partially Proficient 20.9 0.9 21 42 28.6
PD 65.5 1.3 66 69 36.4 Proficient 52.0 2.4 53 65 27.3 Partially Proficient 25.2 1.0 26 52 20.9
PD 85.5 1.6 86 99 10.1 Proficient 57.0 2.1 58 85 49.8 Partially Proficient 25.7 1.7 26 57 26.0
PD 91.4 1.0 92 99 19.8 Proficient 51.1 1.5 52 91 58.5 Partially Proficient 25.0 0.9 25 51 13.5
PD 120.3 0.8 121 129 7.9 Proficient 83.6 4.3 84 120 33.5 Partially Proficient 36.3 3.2 37 83 39.8
PD 125.9 5.8 126 129 9.5
Proficient 83.2 4.3 84 125 34.4
Partially Proficient 24.1 11.9 25 83 42.3
SBP NA NA 0 24 13.7
Achievement level Average
cut Standard
error Raw
Min
score
Max
Percent of students
PD 35.5 5.9 36 46 17.2 Proficient 31.0 6.4 32 35 11.1 Partially Proficient 11.4 1.9 12 31 69.2
PD 45.7 0.1 46 46 7.7 Proficient 30.2 1.6 31 45 33.7 Partially Proficient 18.8 0.9 19 30 33.2
PD 46.0 1.3 NA NA NA Proficient 29.3 1.7 30 46 65.7
continued
Grade
Table 3-1. 2010 PAAP Standard Setting: Round 1 Mathematics
2
SBP NA NA 0 24 25.6
3
SBP NA NA 0 25 22.9
4
SBP NA NA 0 20 8.5
5
SBP NA NA 0 25 15.5
6
SBP NA NA 0 25 14.1
7
SBP NA NA 0 24 8.2
10
SBP NA NA 0 36 18.8
11
Grade
Table 3-2. 2010 PAAP Standard Setting: Round 1 Reading
2
SBP NA NA 0 11 2.5
3
SBP NA NA 0 18 25.5
4
Tasks During Standard Setting 8 2010 Maine PAAP Standard Setting Report
Achievement level Average cut
Standard error
Raw
Min score
Max Percent of students
Partially Proficient 14.9 1.2 15 29 25.9
PD 43.3 1.0 44 46 42.3 Proficient 24.5 1.1 25 43 38.9 Partially Proficient 12.6 0.6 13 24 15.4
PD 61.9 0.9 62 66 5.1 Proficient 50.7 1.6 51 61 31.2 Partially Proficient 31.6 1.4 32 50 33.3
PD 63.1 0.6 64 66 24.8 Proficient 32.8 1.6 33 63 51.8 Partially Proficient 17.1 0.7 18 32 14.7
PD 85.5 0.3 86 86 1.6 Proficient 65.8 5.5 66 85 27.7 Partially Proficient 21.8 2.7 22 65 53.7
PD 87.1 2.1 NA NA NA Proficient 56.2 2.3 57 86 45.9 Partially Proficient 32.5 1.2 33 56 30.6 SBP NA NA 0 32 23.6
Achievement level Average
cut Standard
error Raw
Min
score
Max
Percent of students
PD 64.7 1.5 65 69 19.3 Proficient 44.2 1.4 45 64 44.8 Partially Proficient 24.0 0.6 24 44 22.1
PD 91.9 1.5 92 99 9.1 Proficient 57.5 2.2 58 91 58.9 Partially Proficient 29.2 1.8 30 57 23.3
PD 128.2 4.3 129 129 2.6 Proficient 88.6 6.9 89 128 30.7 Partially Proficient 46.6 3.4 47 88 43.9 SBP NA NA 0 46 22.8
Grade
4
SBP NA NA 0 14 8.4
5
SBP NA NA 0 12 3.4
6
SBP NA NA 0 31 30.3
7
SBP NA NA 0 17 8.7
10
SBP NA NA 0 21 17.0
11
Grade
Table 3-3. 2010 PAAP Standard Setting: Round 1 Science
5
SBP NA NA 0 23 13.8
8
SBP NA NA 0 29 8.7
11
Table 3-4. 2010 PAAP Standard Setting: Round 1 Writing
Grade
Achievement level Average
cut Standard
error Raw
Min
score
Max
Percent of students
PD 23.0 0.2 23 23 16.2 4 Proficient 16.2 0.7 17 22 29.9
Partially Proficient 8.2 0.5 9 16 24.9
continued
Tasks During Standard Setting 9 2010 Maine PAAP Standard Setting Report
Grade Achievement level Average cut
Standard error
Raw
Min score
Max Percent of students
4 SBP NA NA 0 8 29.0
PD 32.8 0.1 33 33 11.7
7 Proficient 15.7 0.6 16 32 60.2
Partially Proficient 11.4 0.2 12 15 14.6 SBP NA NA 0 11 13.6
PD 42.8 2.2 43 43 4.3
11 Proficient 19.5 3.9 20 42 56.0 Partially Proficient 8.8 2.2 9 19 24.1 SBP NA NA 0 8 15.5
3.8 Round 2 Judgments
The purpose of Round 2 was for panelists to discuss their Round 1 judgments as a group and determine
whether any revisions were necessary. A psychometrician shared the average cutpoint locations with the
panelists to help inform their group discussion and Round 2 ratings. It is important to note that although the
impact data and raw score ranges are presented in Tables 3-1 through 3-4, they were not shared with the
panelists after Round 1. Prior to the group discussion, the facilitator asked for a show of hands to determine the
number of panelists who had placed each portfolio into each achievement level; the facilitator then recorded
the results on chart paper. Starting with the first portfolio they disagreed on, the panelists began discussing the
categorization of the portfolios according to their initial ratings in the context of the classifications made by
other members of the group. Panelists were encouraged to share their own points of view as well as listen to
the comments of their colleagues. Facilitators made sure the panelists knew that the purpose of the discussion
was not to reach consensus; at every point throughout the standard setting process, panelists were asked to
provide their own best judgment. Once the discussions were complete, the panelists completed the Round 2
rating form.
3.9 Tabulation of Round 2 Results
When Round 2 ratings were complete, the Measured Progress data analysis team calculated the
average cutpoints for the room and associated impact data. The results of the panelists’ Round 2 ratings are
outlined in Tables 3-5 through 3-8.
Table 3-5. 2010 PAAP Standard Setting: Round 2 Mathematics
Grade Achievement level Average
cut
Standard Raw score
error Min Max
Percent of students
PD 62.6 1.2 63 69 3.4
2 Proficient 44.7 1.2 45 62 22.7 Partially Proficient 23.0 1.9 24 44 54.0
SBP NA NA 0 23 19.9
continued
Tasks During Standard Setting 10 2010 Maine PAAP Standard Setting Report
Achievement level Average cut
Standard error
Raw
Min score
Max Percent of students
PD 71.6 1.9 NA NA NA Proficient 51.5 1.2 52 69 37.6 Partially Proficient 25.6 1.4 26 51 39.5
PD 68.1 0.0 69 69 7.3 Proficient 45.7 1.6 46 68 51.7 Partially Proficient 21.5 0.0 22 45 32.1
PD 66.3 0.9 67 69 33.2 Proficient 51.2 0.4 52 66 31.0 Partially Proficient 27.0 0.0 27 51 19.3
PD 85.0 1.2 86 99 10.1 Proficient 55.5 1.8 56 85 52.0 Partially Proficient 25.2 0.5 26 55 23.8
PD 91.3 0.9 92 99 19.8 Proficient 49.9 1.0 50 91 58.5 Partially Proficient 24.9 0.6 25 49 13.5
PD 119.2 0.9 120 129 7.9 Proficient 87.9 2.3 88 119 31.9 Partially Proficient 38.0 2.3 38 87 40.8
PD 122.4 1.8 123 129 10.8 Proficient 77.1 1.0 78 122 39.0 Partially Proficient 31.8 3.0 32 77 34.0 SBP NA NA 0 31 16.2
Achievement level Average
cut Standard
error Raw
Min
score
Max
Percent of students
PD 45.2 0.1 46 46 4.5 Proficient 33.0 0.3 33 45 23.7 Partially Proficient 18.0 0.3 19 32 49.5
PD 45.5 0.1 46 46 7.7 Proficient 29.1 0.7 30 45 39.9 Partially Proficient 18.0 0.4 19 29 26.9
PD 42.8 0.8 43 46 21.5 Proficient 23.7 0.7 24 42 55.8 Partially Proficient 12.0 0.5 12 23 20.3
Grade
3
SBP NA NA 0 25 22.9
4
SBP NA NA 0 21 9.0
5
SBP NA NA 0 26 16.6
6
SBP NA NA 0 25 14.1
7
SBP NA NA 0 24 8.2
10
SBP NA NA 0 37 19.4
11
Grade
Table 3-6. 2010 PAAP Standard Setting: Round 2 Reading
2
SBP NA NA 0 18 22.2
3
SBP NA NA 0 18 25.5
4
SBP NA NA 0 11 2.4
5 PD 41.5 0.4 42 46 42.3 Proficient 23.9 0.3 24 41 41.3
continued
Tasks During Standard Setting 11 2010 Maine PAAP Standard Setting Report
Achievement level Average cut
Standard error
Raw
Min score
Max Percent of students
Partially Proficient 13.0 0.3 13 23 13.0
PD 60.6 0.6 61 66 7.7 Proficient 46.6 0.8 47 60 35.5 Partially Proficient 28.2 0.6 29 46 30.3
PD 62.9 0.7 63 66 25.7 Proficient 34.5 1.0 35 62 49.1 Partially Proficient 16.6 0.6 17 34 17.0
PD 85.5 0.0 86 86 1.6 Proficient 54.7 0.8 55 85 48.4 Partially Proficient 29.8 0.3 30 54 27.7
PD 84.5 0.0 85 86 5.0 Proficient 57.4 0.4 58 84 40.9 Partially Proficient 35.2 0.8 36 57 27.3 SBP NA NA 0 35 26.9
Achievement level Average cut
Standard error
Raw
Min score
Max Percent of students
PD 65.0 1.2 65 69 19.3 Proficient 44.0 0.7 45 64 44.8 Partially Proficient 23.5 0.5 24 44 22.1
PD 92.6 0.7 93 99 8.7 Proficient 58.4 0.4 59 92 58.9 Partially Proficient 32.9 0.6 33 58 23.3
PD 126.1 1.2 127 129 3.5 Proficient 86.5 1.3 87 126 32.0 Partially Proficient 49.8 0.8 50 86 34.2 SBP NA NA 0 49 30.3
Grade
5
SBP NA NA 0 12 3.4
6
SBP NA NA 0 28 26.5
7
SBP NA NA 0 16 8.3
10
SBP NA NA 0 29 22.3
11
Grade
Table 3-7. 2010 PAAP Standard Setting: Round 2 Science
5
SBP NA NA 0 23 13.8
8
SBP NA NA 0 32 9.1
11
Table 3-8. 2010 PAAP Standard Setting: Round 2 Writing
Grade Achievement level Average
cut Standard
error Raw
Min
score
Max
Percent of students
PD 22.9 0.0 23 23 16.2
4 Proficient 14.6 0.2 15 22 32.4 Partially Proficient 9.7 0.3 10 14 22.4
SBP NA NA 0 9 29.0
PD 32.9 0.1 33 33 11.7 7 Proficient 15.9 0.3 16 32 60.2
Partially Proficient 11.6 0.1 12 15 14.6
continued
Tasks During Standard Setting 12 2010 Maine PAAP Standard Setting Report
Grade Achievement level Average
cut
Standard Raw score error Min Max
Percent of students
7 SBP NA NA 0 11 13.6
PD 42.1 0.7 43 43 4.3
11 Proficient 24.3 0.5 25 42 37.5 Partially Proficient 11.2 0.6 12 24 42.2
SBP NA NA 0 11 15.9
3.10 Round 3 Judgments
The purpose of Round 3 was for panelists to discuss their Round 2 ratings as a whole group and, if
necessary, to revise their judgments. Prior to the group discussion, the facilitator once again asked for a show
of hands to determine the number of panelists who had placed each portfolio into each achievement level; the
facilitator recorded the results on chart paper. The group average cuts based on the Round 2 results were
presented. In addition, in this round, the group was presented with the impact data. The psychometrician
presented the group average cuts and impact data to the group and explained how to use the information as
they completed their Round 3 discussions. Panelists were encouraged to discuss whether the percentage of
students classified in each performance level seemed reasonable, given their perceptions of the students and
the knowledge, skills, and abilities demonstrated in the portfolios. As in Round 2, starting with the first
portfolio for which there was disagreement, the panelists discussed their ratings, with the impact data
considered as additional context for the discussion. Finally, after the discussions were complete, panelists were
given a final opportunity to revise their ratings. Once again, the facilitator reminded the panelists that they
should use their individual best judgment and that it was not necessary for them to reach consensus.
3.11 Tabulation of Round 3 Results
When Round 3 ratings were complete, the Measured Progress data analysis team once again
calculated the average cutpoints for the room and associated impact data. The results of the panelists’ Round
3 ratings are outlined in Tables 3-9 through 3-12 and in Figures 3-1 through 3-4.
Table 3-9. 2010 PAAP Standard Setting: Round 3 Mathematics
Grade Achievement level Average
cut Standard
error Raw
Min
score
Max
Percent of students
PD 64.0 0.9 64 69 3.4
2 Proficient 43.5 0.9 44 63 26.7 Partially Proficient 23.5 1.5 24 43 50.0
SBP NA NA 0 23 19.9
PD 70.3 1.9 NA NA NA 3 Proficient 51.1 1.2 52 69 37.6
Partially Proficient 26.8 1.2 27 51 38.0
continued
Tasks During Standard Setting 13 2010 Maine PAAP Standard Setting Report
Grade Achievement level Average cut
Standard error
Raw
Min
score
Max
Percent of students
3 SBP NA NA 0 26 24.4
PD 61.1 0.4 62 69 20.5
4 Proficient 41.2 0.3 42 61 42.3
Partially Proficient 21.5 0.0 22 41 28.2 SBP NA NA 0 21 9.0
PD 66.9 0.6 67 69 33.2
5 Proficient 51.7 0.3 52 66 31.0
Partially Proficient 26.8 0.2 27 51 19.3
SBP NA NA 0 26 16.6
PD 84.8 1.2 85 99 10.1
6 Proficient 55.9 1.5 56 84 52.0
Partially Proficient 24.4 0.5 25 55 24.7
SBP NA NA 0 24 13.2
PD 91.3 0.9 92 99 19.8
7 Proficient 49.9 1.0 50 91 58.5
Partially Proficient 24.9 0.6 25 49 13.5
SBP NA NA 0 24 8.2
PD 120.1 1.3 121 129 7.9
10 Proficient 85.6 1.6 86 120 32.5
Partially Proficient 35.5 2.4 36 85 41.4
SBP NA NA 0 35 18.3
PD 122.4 1.8 123 129 10.8
11 Proficient 77.5 0.6 78 122 39.0 Partially Proficient 29.4 2.3 30 77 34.4 SBP NA NA 0 29 15.8
Tasks During Standard Setting 14 2010 Maine PAAP Standard Setting Report
Achievement level Average
cut Standard
error Raw
Min
score
Max
Percent of students
PD 45.2 0.1 46 46 4.5 Proficient 31.7 0.7 32 45 23.7 Partially Proficient 17.7 0.3 18 31 49.5
PD 45.5 0.1 46 46 7.7 Proficient 28.8 0.6 29 45 42.3 Partially Proficient 17.4 0.3 18 28 25.0
PD 42.2 0.2 43 46 21.5 Proficient 24.4 0.3 25 42 54.2 Partially Proficient 11.5 0.4 12 24 21.9
PD 41.6 0.4 42 46 42.3 Proficient 23.9 0.2 24 41 41.3 Partially Proficient 13.0 0.3 14 23 12.0
PD 59.1 1.0 60 66 8.5 Proficient 30.8 0.8 31 59 62.8 Partially Proficient 18.9 0.7 19 30 15.0
PD 64.0 0.5 64 66 24.8 Proficient 37.3 1.5 38 63 44.0 Partially Proficient 17.1 0.7 18 37 22.5
PD 85.5 0.0 86 86 1.6 Proficient 55.6 0.0 56 85 45.7 Partially Proficient 29.5 0.0 30 55 30.3
PD 84.5 0.0 85 86 5.0 Proficient 56.8 0.0 57 84 40.9 Partially Proficient 36.5 0.0 37 56 25.2 SBP NA NA 0 36 28.9
Grade
Table 3-10. 2010 PAAP Standard Setting: Round 3 Reading
2
SBP NA NA 0 17 22.2
3
SBP NA NA 0 17 25.0
4
SBP NA NA 0 11 2.4
5
SBP NA NA 0 13 4.3
6
SBP NA NA 0 18 13.7
7
SBP NA NA 0 17 8.7
10
SBP NA NA 0 29 22.3
11
Tasks During Standard Setting 15 2010 Maine PAAP Standard Setting Report
Achievement level Average
cut Standard
error Raw
Min
score
Max
Percent of students
PD 65.9 1.3 66 69 19.3 Proficient 44.6 0.7 45 65 44.8 Partially Proficient 23.5 0.5 24 44 22.1
PD 92.3 0.7 93 99 8.7 Proficient 58.0 0.5 58 92 59.4 Partially Proficient 32.9 0.6 33 57 22.8
PD 126.1 1.2 127 129 3.5 Proficient 86.5 1.3 87 126 32.0 Partially Proficient 49.1 1.4 50 86 34.2 SBP NA NA 0 49 30.3
Achievement level Average
cut Standard
error Raw
Min
score
Max
Percent of students
PD 23.0 0.0 23 23 16.2 Proficient 14.4 0.1 15 22 32.4 Partially Proficient 9.6 0.1 10 14 22.4
PD 32.9 0.1 33 33 11.7 Proficient 16.2 0.4 17 32 59.2 Partially Proficient 11.6 0.1 12 16 15.5
PD 40.9 0.7 41 43 8.2 Proficient 23.8 0.6 24 40 33.6 Partially Proficient 12.1 0.4 13 23 41.8 SBP NA NA 0 12 16.4
Grade
Table 3-11. 2010 PAAP Standard Setting: Round 3 Science
5
SBP NA NA 0 23 13.8
8
SBP NA NA 0 32 9.1
11
Grade
Table 3-12. 2010 PAAP Standard Setting: Round 3 Writing
4
SBP NA NA 0 9 29.0
7
SBP NA NA 0 11 13.6
11
Tasks During Standard Setting 16 2010 Maine PAAP Standard Setting Report
ro 0
c
0
c
0
Partially Proficient
Figure 3-1. 2010 PAAP Standard Setting: Mathematics: Round 3 Results
0 0.....-
0co
C" 0 0) Q.)
+""'
u CD
c
CJ) +""'
Q.)
""C :::J
+""' CJ) "¢
'+- 0
+""'
Q.)
u I-
Q.)
a_ N
0
02 03 04 05 06 07 10 11 Proficient with Distinction
Proficient
Substantially Below Proficient Mathematics: Round 3 Results
Tasks During Standard Setting 17 2010 Maine PAAP Standard Setting Report
0
u
0
Partially Proficient
Figure 3-2. 2010 PAAP Standard Setting: Reading: Round 3 Results
0 0 ........
0ro
0 0)
..Q....
)..
co u ID
·c-
..C...
J..).
c Q)
"0
..:..:.:.J.. 0 CJ) "¢
'+- 0
........ c Q)
Q)
a_ N
0
02 03 04 05 06 07 10 11 Proficient with Distinction
Proficient
Substantially Below Proficient Reading: Round 3 Results
Tasks During Standard Setting 18 2010 Maine PAAP Standard Setting Report
Figure 3-3. 2010 PAAP Standard Setting: Science: Round 3 Results
0 0......
0co
0 0)
..Q....
)..
co 0 (.) CD
·c-
..C...
J..).
c Q)
"0
..:..:.:.J.. 0 CJ) "¢
'+- 0
........ c Q) (.)
Q) 0
11.. N
0
Proficient with Distinction Proficient
Partially Proficient Substantially Below Proficient
05 08 11
Science: Round 3 Results
Tasks During Standard Setting 19 2010 Maine PAAP Standard Setting Report
c
0
c
0
Figure 3-4. 2010 PAAP Standard Setting: Writing: Round 3 Results
0 0) Q)
+
c"
o"'
(.)
c
0 0 .....--
0 CX)
0 <D
(/) +""'
Q)
""0 ::J
+""' (/) "¢
'+- 0
+""'
Q) (.) I-
Q)
a_ C'\1
0
Proficient with Distinction
Proficient Partially Proficient Substantially Below Proficient
04 07 11
Writing: Round 3 Results
Tasks During Standard Setting 20 2010 Maine PAAP Standard Setting Report
3.12 Repeat Process for Second Grade and Content Area
(Mathematics and Reading Only)
As mentioned above, eight panels each recommended cutpoints for two grade levels in mathematics
and reading. For those panels, once they had completed the entire process for the first grade level, they (1)
completed a process evaluation, giving their perceptions of the standard setting process and results thus far,
and (2) repeated the entire process (except for the training evaluation) for the second grade level.
3.13 Evaluation
The measurement literature sometimes considers the evaluation process to be another product of the
standard setting process (e.g., Reckase, 2001). To provide evidence of the participants’ views of the standard
setting process, panelists were asked to complete questionnaires throughout the process. The results of the
evaluations are presented in Appendix I.
Tasks During Standard Setting 21 2010 Maine PAAP Standard Setting Report
Chapter 4. TASKS COMPLETED AFTER STANDARD SETTING
Upon conclusion of the standard setting meeting, several important tasks were completed. These tasks
centered on reviewing the standard setting process and addressing anomalies that may have occurred in the
process or in the outcomes, presenting the results to MDOE and making any final revisions or adjustments.
4.1 Analysis and Review of Panelists’ Feedback
Upon completion of the evaluation forms, panelists’ responses were reviewed. In general, this review
did not reveal any anomalies in the standard setting process or indicate any reason that a particular panelist’s
data should not be included when the final cutpoints were calculated. The one exception was a panelist in
grade 10 mathematics who did not appear to adequately understand the standard setting task. The panelist
continually placed the Proficient cut at a higher achievement than the Proficient With Distinction cut.
Although, the panelist self-corrected this issue for grade 11 mathematics, the panelist’s ratings were removed
from the grade 10 mathematics calculations and are not reflected in the results presented in this report. It
appeared that all remaining panelists understood the rating task and attended to it appropriately.
4.2 Calculation of Recommended Cutpoints
At the end of the standard setting meeting, the Round 3 cuts were presented to the Department of
Education as the final results of the standard setting meeting. Following the standard setting, a few concerns
were raised about the cuts. Most important of these was that higher standards (from a raw score cut
perspective) were established by the panelists in grades 2 and 3 than in grades 4 and 5 despite the fact that
these grades are based on the same task banks, level of complexity, and raw score scale. In theory, the grade 2
and grade 3 cuts should represent similar achievement requirements as grades 4 and 5. In addition, because a
single task bank is used for High School and because of the population of interest, a single set of cuts for
these students seemed more appropriate. Finally, the Proficient cut was set at a higher level in grade 6
mathematics than in grade 7 mathematics. Given that these two grades are based on the same task banks, level
of complexity, and raw score scale, it was felt that proficiency in the two grades should represent similar
achievement requirements. Consequently, the following adjustments were made to the Round 3 raw score cuts
and are presented in Tables 4-1 through 4-2 and Figures 4-1 through 4-2.
Grade 4 raw score cuts were applied to grade 2 and grade 3.
A single set of cuts was established for High School by averaging the grade 10 and grade 11 raw
score cuts.
The grade 7 Proficient cut in mathematics was raised to equal the grade 6 Proficient cut in
mathematics.
Tasks After Standard Setting 22 2010 Maine PAAP Standard Setting Report
Achievement level Average
cut Standard
error Raw
Min
score
Max
Percent of students
PD 61.1 NA 62 69 3.4 Proficient 41.2 NA 42 61 30.7 Partially Proficient 21.5 NA 22 41 58
PD 61.1 NA 62 69 14.1 Proficient 41.2 NA 42 61 34.6 Partially Proficient 21.5 NA 22 41 40
PD 61.1 0.4 62 69 20.5 Proficient 41.2 0.3 42 61 42.3 Partially Proficient 21.5 0 22 41 28.2
PD 66.9 0.6 67 69 33.2 Proficient 51.7 0.3 52 66 31 Partially Proficient 26.8 0.2 27 51 19.3
PD 84.8 1.2 85 99 10.1 Proficient 55.9 1.5 56 84 52 Partially Proficient 24.4 0.5 25 55 24.7
PD 91.3 0.9 92 99 19.8 Proficient 55.9 NA 56 91 53.1 Partially Proficient 24.9 0.6 25 55 18.8
PD 121.25 NA 122 129 9 Proficient 81.55 NA 82 121 35.4 Partially Proficient 32.45 NA 33 81 38.4 SBP NA NA 0 32 17.1
Grade
Table 4-1. 2010 PAAP Standard Setting: Mathematics: Adjusted Results
2
SBP NA NA 0 21 8
3
SBP NA NA 0 21 11.2
4
SBP NA NA 0 21 9
5
SBP NA NA 0 26 16.6
6
SBP NA NA 0 24 13.2
7
SBP NA NA 0 24 8.2
HS
Tasks After Standard Setting 23 2010 Maine PAAP Standard Setting Report
Achievement level Average
cut Standard
error Raw
Min
score
Max
Percent of students
PD 42.2 NA 43 46 7.6 Proficient 24.4 NA 25 42 53 Partially Proficient 11.5 NA 12 24 36.9
PD 42.2 NA 43 46 17.3 Proficient 24.4 NA 25 42 45.2 Partially Proficient 11.5 NA 12 24 34.6
PD 42.2 0.2 43 46 21.5 Proficient 24.4 0.3 25 42 54.2 Partially Proficient 11.5 0.4 12 24 21.9
PD 41.6 0.4 42 46 42.3 Proficient 23.9 0.2 24 41 41.3 Partially Proficient 13 0.3 14 23 12
PD 59.1 1 60 66 8.5 Proficient 30.8 0.8 31 59 62.8 Partially Proficient 18.9 0.7 19 30 15
PD 64 0.5 64 66 24.8 Proficient 37.3 1.5 38 63 44 Partially Proficient 17.1 0.7 18 37 22.5
PD 85 NA 85 86 4.2 Proficient 56.2 NA 57 84 40.2 Partially Proficient 33 NA 33 56 30.9 SBP NA NA 0 32 24.7
Grade
Table 4-2. 2010 PAAP Standard Setting: Reading: Adjusted Results
2
SBP NA NA 0 11 2.5
3
SBP NA NA 0 11 2.9
4
SBP NA NA 0 11 2.4
5
SBP NA NA 0 13 4.3
6
SBP NA NA 0 18 13.7
7
SBP NA NA 0 17 8.7
HS
Tasks After Standard Setting 24 2010 Maine PAAP Standard Setting Report
ro 0
0
0
Figure 4-1. 2010 PAAP Standard Setting: Mathematics: Adjusted Results
0 0......
0co
C" 0 0) Q.)
+""'
u c.o c
CJ) +""'
Q.)
""C :::J
+""' CJ) 'V
'+- 0
+""'
Q.)
u I-
Q.)
a_ N
0
2 3 4 5 6 7 HS Proficient with Distinction
Proficient
Partially Proficient Substantially Below Proficient
Mathematics: Adjusted Results
Tasks After Standard Setting 25 2010 Maine PAAP Standard Setting Report
ro 0
c
0
c
0
Figure 4-2. 2010 PAAP Standard Setting: Reading: Adjusted Results
0 0.....-
0 CX)
C" 0 0) Q.)
+""'
u CD
c
CJ) +""'
Q.)
""C :::J
+""' CJ) "¢
'+- 0
+""'
Q.)
u I-
Q.)
a_ N
0
2 3 4 5 6 7 HS Proficient with Distinction
Proficient
Partially Proficient Substantially Below Proficient
Reading: Adjusted Results
Tasks After Standard Setting 26 2010 Maine PAAP Standard Setting Report
After carefully considering the above information, the MDOE remained concerned about a few of the
cut scores in mathematics and writing. Consequently, the department decided to make a final policy adjustment
to some of the cut scores. The resulting DOE-approved operational cut scores, and a more detailed explanation
outlining how the final cut scores were established can be found in Appendix J.
4.3 Preparation of this Standard Setting Report
Following final compilation of standard setting results, Measured Progress prepared this report, which
documents the procedures and results of the 2010 standard setting meeting in order to establish performance
standards for the Maine PAAP in mathematics, reading, science, and writing.
Tasks After Standard Setting 27 2010 Maine PAAP Standard Setting Report
REFERENCES
American Educational Research Association, American Psychological Association, & National Council on
Measurement in Education (1999). Standards for educational and psychological testing. Washington,
DC: American Educational Research Association.
Cizek, G. J., & Bunch, M. B. (2007). Standard setting: Establishing and evaluating performance standards
on tests. Thousand Oaks, CA: Sage Publications.
Kingston, N., Kahl, S., Sweeney, K., & Bay, L. (2001). Setting performance standards using the body of
work method. Setting performance standards: concepts, methods, and perspectives (pp. 219–
248). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.
Reckase, M.D. (2001). Innovative methods for helping standard-setting participants to perform their task:
The role of feedback regarding consistency, accuracy, and impact. In G. J. Cizek (Ed.) Setting
performance standards: concepts, methods, and perspectives (pp. 159-173). Mahwah, NJ: Lawrence
Erlbaum.
U.S. Department of Education (2009). Standards and assessments peer review guidance: Information and
examples for meeting requirements of the No Child Left Behind Act of 2001. Washington, DC: U.S.
Department of Education Office of Elementary and Secondary Education. Retrieved June 10, 2010
from the World Wide Web: www2.ed.gov/policy/elsec/guid/saaprguidance.pdf.
References 28 2010 Maine PAAP Standard Setting Report
APPENDICES
Appendices 29 2010 Maine PAAP Standard Setting Report
Appendix A—ACHIEVEMENT LEVEL DEFINITIONS
Appendix A—Achievement Level Definitions 1 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Reading ~ Grade 2
Reading achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate reading standards. These definitions serve as the
foundation for achievement level definitions for the alternate grade level expectations in reading.
Skills associated with these achievement level definitions may include some of the following:
• student initiated communication
through signs, symbols, pictures,
gestures, and/or oral language
• relating of symbols to the
objects/ideas they represent
• using phonemic awareness skills
• using context clues to determine
meaning of words
• reading aloud sight words
• putting key events in correct
sequence
• identifying setting or characters
• comprehension of information
gained through listening or
viewing
• retelling events in a story
Appropriate performance at grade two is clarified by the level of complexity within the standard.
Proficient with Distinction The student’s performance demonstrates the skillful ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the
consistent ability to read, comprehend, and interpret literary OR informational texts appropriate
to the student’s instructional level.
Proficient The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also generally demonstrates the
ability to read, comprehend, and interpret literary OR informational texts appropriate to the
student’s instructional level.
Partially Proficient The student’s performance demonstrates the inconsistent ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also
demonstrates the incomplete or inconsistent ability to read, comprehend, and interpret literary
OR informational texts appropriate to the student’s instructional level.
Substantially Below Proficient The student’s performance demonstrates the limited ability to identify and decode unfamiliar vocabulary and/or demonstrates incorrect understanding of word meaning. The work also
demonstrates minimal or limited ability to read, comprehend, and interpret literary OR
informational texts appropriate to the student’s instructional level.
Appendix A—Achievement Level Definitions 3 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Reading ~ Grade 3
Reading achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate reading standards. These definitions serve as the
foundation for achievement level definitions for the alternate grade level expectations in reading.
Skills associated with these achievement level definitions may include some of the following:
• student initiated communication
through signs, symbols, pictures,
gestures, and/or oral language
• relating of symbols to the
objects/ideas they represent
• using phonemic awareness skills
• using context clues to determine
meaning of words
• reading aloud sight words
• distinguishing parts of a book
(e.g., front, top, bottom, title,
author)
• using explicitly stated information
from the text to answer questions
• recognizing a central idea from
text when presented with pictures
Appropriate performance at grade three is clarified by the level of complexity within the
standard.
Proficient with Distinction The student’s performance demonstrates the skillful ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the
consistent ability to read, comprehend, and interpret literary OR informational texts appropriate
to the student’s instructional level.
Proficient The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also generally demonstrates the
ability to read, comprehend, and interpret literary OR informational texts appropriate to the
student’s instructional level.
Partially Proficient The student’s performance demonstrates the inconsistent ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also
demonstrates the incomplete or inconsistent ability to read, comprehend, and interpret literary
OR informational texts appropriate to the student’s instructional level.
Substantially Below Proficient The student’s performance demonstrates the limited ability to identify and decode unfamiliar vocabulary and/or demonstrates incorrect understanding of word meaning. The work also
demonstrates minimal or limited ability to read, comprehend, and interpret literary OR
informational texts appropriate to the student’s instructional level.
Appendix A—Achievement Level Definitions 4 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Reading ~ Grade 4
Reading achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate reading standards. These definitions serve as the
foundation for achievement level definitions for the alternate grade level expectations in reading.
Skills associated with these achievement level definitions may include some of the following:
• student initiated communication
through signs, symbols, pictures,
gestures, and/or oral language
• relating of symbols to the
objects/ideas they represent
• using phonemic awareness skills
• using context clues to determine
meaning of words
• reading aloud sight words
• putting key events in correct
sequence
• identifying setting or characters
• comprehension of information
gained through listening or
viewing
• retelling events in a story
Appropriate performance at grade four is clarified by the level of complexity within the standard.
Proficient with Distinction The student’s performance demonstrates the skillful ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the
consistent ability to read, comprehend, and interpret literary OR informational texts appropriate
to the student’s instructional level.
Proficient The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also generally demonstrates the
ability to read, comprehend, and interpret literary OR informational texts appropriate to the
student’s instructional level.
Partially Proficient The student’s performance demonstrates the inconsistent ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also
demonstrates the incomplete or inconsistent ability to read, comprehend, and interpret literary
OR informational texts appropriate to the student’s instructional level.
Substantially Below Proficient The student’s performance demonstrates the limited ability to identify and decode unfamiliar vocabulary and/or demonstrates incorrect understanding of word meaning. The work also
demonstrates minimal or limited ability to read, comprehend, and interpret literary OR
informational texts appropriate to the student’s instructional level.
Appendix A—Achievement Level Definitions 5 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Reading ~ Grade 5
Reading achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate reading standards. These definitions serve as the
foundation for achievement level definitions for the alternate grade level expectations in reading.
Skills associated with these achievement level definitions may include some of the following:
• student initiated communication
through signs, symbols, pictures,
gestures, and/or oral language
• relating of symbols to the
objects/ideas they represent
• using phonemic awareness skills
• using context clues to determine
meaning of words
• reading aloud sight words
• distinguishing parts of a book
(e.g., front, top, bottom, title,
author)
• using explicitly stated information
from the text to answer questions
• recognizing a central idea from
text when presented with pictures
Appropriate performance at grade five is clarified by the level of complexity within the standard.
Proficient with Distinction The student’s performance demonstrates the skillful ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the
consistent ability to read, comprehend, and interpret literary OR informational texts appropriate
to the student’s instructional level.
Proficient The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also generally demonstrates the
ability to read, comprehend, and interpret literary OR informational texts appropriate to the
student’s instructional level.
Partially Proficient The student’s performance demonstrates the inconsistent ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also
demonstrates the incomplete or inconsistent ability to read, comprehend, and interpret literary
OR informational texts appropriate to the student’s instructional level.
Substantially Below Proficient The student’s performance demonstrates the limited ability to identify and decode unfamiliar vocabulary and/or demonstrates incorrect understanding of word meaning. The work also
demonstrates minimal or limited ability to read, comprehend, and interpret literary OR
informational texts appropriate to the student’s instructional level.
Appendix A—Achievement Level Definitions 6 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Reading ~ Grade 6
Reading achievement level definitions describe the quality of a student’s responses on state-level alternate
assessments in relation to the alternate reading standards. These definitions serve as the foundation for
achievement level definitions for the alternate grade level expectations in reading.
Skills associated with these achievement level definitions may include some of the following:
• relating of symbols to the objects/ideas
they represent • using phonemic awareness skills
• using context clues, a dictionary and/or
glossary to determine meaning of words
• using word parts, phonics, knowledge
of sounds, syllable types and/or word
patterns to decode
• identifying synonyms, antonyms and/or
categorizing words
• reading aloud sight words
• using explicitly stated information
from the text to answer questions • recognizing a central idea from text
when presented with pictures
• identifying or describing characters,
setting, problems, solutions, events
and/or plot • making logical predictions
• making basic inferences
• paraphrasing or summarizing
Appropriate performance at grade six is clarified by the level of complexity within the standard.
Proficient with Distinction
The student’s performance demonstrates the skillful ability to identify and decode unfamiliar vocabulary
and/or demonstrate understanding of word meaning. The work also demonstrates the consistent ability to
read, comprehend, and interpret literary OR informational texts appropriate to the student’s instructional
level.
Proficient The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary and/or
demonstrate understanding of word meaning. The work also generally demonstrates the ability to read,
comprehend, and interpret literary OR informational texts appropriate to the student’s instructional level.
Partially Proficient The student’s performance demonstrates the inconsistent ability to identify and decode unfamiliar
vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the
incomplete or inconsistent ability to read, comprehend, and interpret literary OR informational texts
appropriate to the student’s instructional level.
Substantially Below Proficient
The student’s performance demonstrates the limited ability to identify and decode unfamiliar vocabulary
and/or demonstrates incorrect understanding of word meaning. The work also demonstrates minimal or
limited ability to read, comprehend, and interpret literary OR informational texts appropriate to the
student’s instructional level.
Appendix A—Achievement Level Definitions 7 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Reading ~ Grade 7
Reading achievement level definitions describe the quality of a student’s responses on state-level alternate
assessments in relation to the alternate reading standards. These definitions serve as the foundation for
achievement level definitions for the alternate grade level expectations in reading.
Skills associated with these achievement level definitions may include some of the following:
• relating of symbols to the objects/ideas
they represent • using phonemic awareness skills
• using context clues, a dictionary and/or
glossary to determine meaning of words
• using word parts, phonics, knowledge
of sounds, syllable types and/or word
patterns to decode
• identifying synonyms, antonyms and/or
categorizing words
• reading aloud sight words
• using explicitly stated information from
the text to answer questions
• recognizing a central idea from text
when presented with pictures • obtaining information from the table
of contents, glossary, table of contents
• connecting information within a text
• identifying parts of a book (e.g.,
author, title, beginning, end)
• Paraphrasing or summarizing
• Drawing conclusions or making
inferences
Appropriate performance at grade seven is clarified by the level of complexity within the standard.
Proficient with Distinction The student’s performance demonstrates the skillful ability to identify and decode unfamiliar vocabulary
and/or demonstrate understanding of word meaning. The work also demonstrates the consistent ability to
read, comprehend, and interpret literary OR informational texts appropriate to the student’s instructional
level.
Proficient The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary and/or
demonstrate understanding of word meaning. The work also generally demonstrates the ability to read,
comprehend, and interpret literary OR informational texts appropriate to the student’s instructional level.
Partially Proficient The student’s performance demonstrates the inconsistent ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the
incomplete or inconsistent ability to read, comprehend, and interpret literary OR informational texts
appropriate to the student’s instructional level.
Substantially Below Proficient The student’s performance demonstrates the limited ability to identify and decode unfamiliar vocabulary
and/or demonstrates incorrect understanding of word meaning. The work also demonstrates minimal or
limited ability to read, comprehend, and interpret literary OR informational texts appropriate to the
student’s instructional level.
Appendix A—Achievement Level Definitions 8 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Reading ~ High School
Reading achievement level definitions describe the quality of a student’s responses on state-level alternate
assessments in relation to the alternate reading standards. These definitions serve as the foundation for
achievement level definitions for the alternate grade level expectations in reading.
Skills associated with these achievement level definitions may include some of the following:
• using phonemic awareness skills
• using context clues, dictionary, glossary
and/or thesaurus to determine meaning
of words
• using word parts or phonics to decode • reading aloud sight words
• identifying synonyms, antonyms and/or
categorizing words
• using explicitly stated information from
the text to answer questions
• recognizing a central idea from text
when presented with pictures
• identifying or describing characters,
setting, problems, solutions and/or
events
• identifying author’s basic message
and/or purpose • identifying narrator
• paraphrasing, summarizing, and/or
comparing/contrasting
• making logical predictions
• making inferences
• recognizing a central idea from text
when presented with pictures • obtaining information from the table
of contents or glossary
• connecting information within a text • synthesizing information
Appropriate performance at high school level is clarified by the level of complexity within the standard.
Proficient with Distinction The student’s performance demonstrates the skillful ability to identify and decode unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the consistent ability to
read, comprehend, and interpret literary OR informational texts appropriate to the student’s instructional
level.
Proficient
The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary and/or
demonstrate understanding of word meaning. The work also generally demonstrates the ability to read,
comprehend, and interpret literary OR informational texts appropriate to the student’s instructional level.
Partially Proficient
The student’s performance demonstrates the inconsistent ability to identify and decode unfamiliar
vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the
incomplete or inconsistent ability to read, comprehend, and interpret literary OR informational texts
appropriate to the student’s instructional level.
Substantially Below Proficient The student’s performance demonstrates the limited ability to identify and decode unfamiliar vocabulary and/or demonstrates incorrect understanding of word meaning. The work also demonstrates minimal or
limited ability to read, comprehend, and interpret literary OR informational texts appropriate to the
student’s instructional level.
Appendix A—Achievement Level Definitions 9 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Mathematics ~ Grade 2
Mathematics achievement level definitions describe the quality of a student’s responses on statelevel
alternate assessments in relation to the alternate mathematics standards. These definitions
serve as the foundation for achievement level definitions for the alternate grade level
expectations in mathematics.
Skills associated with these achievement level definitions may include some of the following:
• matching small collections of equivalent sets
• matching simple 2-D shapes
• comparing two items based on multiple attributes
• identifying measurement tools
• copying simple patterns
• collecting data
• identifying the outcome of an event
Appropriate performance at grade 2 is clarified by the level of complexity within the standard.
Proficient with Distinction The student’s work demonstrates an understanding of essential concepts in mathematics, including the ability to make connections among central ideas. The student’s responses
demonstrate the ability to utilize information and solve problems including implementing
strategies, accurately performing procedures and providing solutions.
Proficient The student’s work demonstrates an understanding of basic concepts in mathematics and connections among central ideas. The student’s responses demonstrate basic ability to solve
problems, including performing procedures and providing solutions. The student’s work may
contain minor errors.
Partially Proficient The student’s work demonstrates an incomplete understanding of basic concepts in mathematics and inconsistent connections among central ideas. The student’s responses demonstrate limited
ability to solve problems, including performing procedures and providing solutions. Problem
solving strategies may be flawed and procedures preformed inaccurately.
Substantially Below Proficient The student’s work demonstrates limited understanding of basic concepts in mathematics and inaccurate connections among central ideas. The student’s responses demonstrate minimal ability
to solve problems. Problem solving strategies may be flawed or inappropriate and there may be
many omissions.
Appendix A—Achievement Level Definitions 10 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Mathematics ~ Grade 3
Mathematics achievement level definitions describe the quality of a student’s responses on statelevel
alternate assessments in relation to the alternate mathematics standards. These definitions
serve as the foundation for achievement level definitions for the alternate grade level
expectations in mathematics.
Skills associated with these achievement level definitions may include some of the following:
• indicating or labeling collections of
equivalent sets
• reading, writing, and counting
numbers
• recognizing place value
• skip counting
• comparing two items based on
multiple attributes
• identifying measurement tools
• copying and/or extending patterns
• collecting data
• identifying the outcome of an
event
Appropriate performance at grade three is clarified by the level of complexity within the
standard.
Proficient with Distinction The student’s work demonstrates an understanding of essential concepts in mathematics, including the ability to make connections among central ideas. The student’s responses
demonstrate the ability to utilize information and solve problems including implementing
strategies, accurately performing procedures and providing solutions.
Proficient The student’s work demonstrates an understanding of basic concepts in mathematics and connections among central ideas. The student’s responses demonstrate basic ability to solve
problems, including performing procedures and providing solutions. The student’s work may
contain minor errors.
Partially Proficient The student’s work demonstrates an incomplete understanding of basic concepts in mathematics and inconsistent connections among central ideas. The student’s responses demonstrate limited
ability to solve problems, including performing procedures and providing solutions. Problem
solving strategies may be flawed and procedures preformed inaccurately.
Substantially Below Proficient The student’s work demonstrates limited understanding of basic concepts in mathematics and inaccurate connections among central ideas. The student’s responses demonstrate minimal ability
to solve problems. Problem solving strategies may be flawed or inappropriate and there may be
many omissions.
Appendix A—Achievement Level Definitions 11 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Mathematics ~ Grade 4
Mathematics achievement level definitions describe the quality of a student’s responses on statelevel
alternate assessments in relation to the alternate mathematics standards. These definitions
serve as the foundation for achievement level definitions for the alternate grade level
expectations in mathematics.
Skills associated with these achievement level definitions may include some of the following:
• ordering and comparing numbers
• identifying two-digit numbers
• solving addition and subtraction
problems involving one-digit
numbers
• identifying 2-D shapes
• comparing two items based on
multiple attributes
• identifying congruent figures
• identifying and using measurement
tools
• copying and extending simple
patterns
• collecting, organizing, and/or
interpreting data
• identifying the outcome of an
event
Appropriate performance at grade four is clarified by the level of complexity within the standard.
Proficient with Distinction The student’s work demonstrates an understanding of essential concepts in mathematics, including the ability to make connections among central ideas. The student’s responses
demonstrate the ability to utilize information and solve problems including implementing
strategies, accurately performing procedures and providing solutions.
Proficient The student’s work demonstrates an understanding of basic concepts in mathematics and connections among central ideas. The student’s responses demonstrate basic ability to solve
problems, including performing procedures and providing solutions. The student’s work may
contain minor errors.
Partially Proficient The student’s work demonstrates an incomplete understanding of basic concepts in mathematics and inconsistent connections among central ideas. The student’s responses demonstrate limited
ability to solve problems, including performing procedures and providing solutions. Problem
solving strategies may be flawed and procedures preformed inaccurately.
Substantially Below Proficient The student’s work demonstrates limited understanding of basic concepts in mathematics and inaccurate connections among central ideas. The student’s responses demonstrate minimal ability
to solve problems. Problem solving strategies may be flawed or inappropriate and there may be
many omissions.
Appendix A—Achievement Level Definitions 12 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Mathematics ~ Grade 5
Mathematics achievement level definitions describe the quality of a student’s responses on statelevel
alternate assessments in relation to the alternate mathematics standards. These definitions
serve as the foundation for achievement level definitions for the alternate grade level
expectations in mathematics.
Skills associated with these achievement level definitions may include some of the following:
• matching and/or identifying coins
• decimal notation
• comparing two items based on multiple attributes
• identifying and using measurement tools
• copying and extending patterns
Appropriate performance at grade five is clarified by the level of complexity within the standard.
Proficient with Distinction The student’s work demonstrates an understanding of essential concepts in mathematics, including the ability to make connections among central ideas. The student’s responses
demonstrate the ability to utilize information and solve problems including implementing
strategies, accurately performing procedures and providing solutions.
Proficient The student’s work demonstrates an understanding of basic concepts in mathematics and connections among central ideas. The student’s responses demonstrate basic ability to solve
problems, including performing procedures and providing solutions. The student’s work may
contain minor errors.
Partially Proficient The student’s work demonstrates an incomplete understanding of basic concepts in mathematics and inconsistent connections among central ideas. The student’s responses demonstrate limited
ability to solve problems, including performing procedures and providing solutions. Problem
solving strategies may be flawed and procedures preformed inaccurately.
Substantially Below Proficient The student’s work demonstrates limited understanding of basic concepts in mathematics and inaccurate connections among central ideas. The student’s responses demonstrate minimal ability
to solve problems. Problem solving strategies may be flawed or inappropriate and there may be
many omissions.
Appendix A—Achievement Level Definitions 13 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Mathematics ~ Grade 6
Mathematics achievement level definitions describe the quality of a student’s responses on statelevel
alternate assessments in relation to the alternate mathematics standards. These definitions
serve as the foundation for achievement level definitions for the alternate grade level
expectations in mathematics.
Skills associated with these achievement level definitions may include some of the following:
• indicating, comparing, and/or ordering rational numbers limited to fractions with
denominators of 2, 3, 4, and/or 5
• identifying and/or classifying 2-D shapes and/or angles
• matching quantities that are equal
• finding the value that will make an open sentence true
Appropriate performance at grade six is clarified by the level of complexity within the standard.
Proficient with Distinction The student’s work demonstrates an understanding of essential concepts in mathematics, including the ability to make connections among central ideas. The student’s responses
demonstrate the ability to utilize information and solve problems including implementing
strategies, accurately performing procedures and providing solutions.
Proficient The student’s work demonstrates an understanding of basic concepts in mathematics and connections among central ideas. The student’s responses demonstrate basic ability to solve
problems, including performing procedures and providing solutions. The student’s work may
contain minor errors.
Partially Proficient The student’s work demonstrates an incomplete understanding of basic concepts in mathematics and inconsistent connections among central ideas. The student’s responses demonstrate limited
ability to solve problems, including performing procedures and providing solutions. Problem
solving strategies may be flawed and procedures preformed inaccurately.
Substantially Below Proficient The student’s work demonstrates limited understanding of basic concepts in mathematics and inaccurate connections among central ideas. The student’s responses demonstrate minimal ability
to solve problems. Problem solving strategies may be flawed or inappropriate and there may be
many omissions.
Appendix A—Achievement Level Definitions 14 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Mathematics ~ Grade 7
Mathematics achievement level definitions describe the quality of a student’s responses on statelevel
alternate assessments in relation to the alternate mathematics standards. These definitions
serve as the foundation for achievement level definitions for the alternate grade level
expectations in mathematics.
Skills associated with these achievement level definitions may include some of the following:
• ordering and comparing whole numbers
• using measurement tools and estimating outcomes
• computing equivalencies
• using more, less, equal, and/or other comparisons to analyze data or solve problems
Appropriate performance at grade seven is clarified by the level of complexity within the
standard.
Proficient with Distinction: The student’s work demonstrates an understanding of essential concepts in mathematics, including the ability to make connections among central ideas. The student’s responses
demonstrate the ability to utilize information and solve problems including implementing
strategies, accurately performing procedures and providing solutions.
Proficient The student’s work demonstrates an understanding of basic concepts in mathematics and connections among central ideas. The student’s responses demonstrate basic ability to solve
problems, including performing procedures and providing solutions. The student’s work may
contain minor errors.
Partially Proficient The student’s work demonstrates an incomplete understanding of basic concepts in mathematics and inconsistent connections among central ideas. The student’s responses demonstrate limited
ability to solve problems, including performing procedures and providing solutions. Problem
solving strategies may be flawed and procedures preformed inaccurately.
Substantially Below Proficient The student’s work demonstrates limited understanding of basic concepts in mathematics and inaccurate connections among central ideas. The student’s responses demonstrate minimal ability
to solve problems. Problem solving strategies may be flawed or inappropriate and there may be
many omissions.
Appendix A—Achievement Level Definitions 15 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Mathematics ~ High School
Mathematics achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate mathematics standards. These definitions serve as the
foundation for achievement level definitions for the alternate grade level expectations in mathematics.
Skills associated with these achievement level definitions may include some of the following:
• solving multi-step addition, subtraction, multiplication and/or division problems
involving whole numbers, fractions,
decimals, percents and/or ratios
• describing or illustrating the
relationships between the four operations
• copying, extending, and describing
patterns • writing rules for finding specific cases
of a linear or nonlinear relationship
• finding the value that will make an open
sentence true
• representing unknown quantities with
letters
• simplifying and writing linear
algebraic expressions • collecting, arranging, interpreting,
and/or analyzing data to formulate or
justify conclusions, make predictions
or solve problems
• using more, less, equal, and/or other
comparisons to analyze data or solve
problems,
• using measures of central tendency or
range to analyze situations or solve problems
Appropriate performance at the high school level is clarified by the level of complexity within the
standard.
Proficient with Distinction The student’s work demonstrates an understanding of essential concepts in mathematics, including the
ability to make connections among central ideas. The student’s responses demonstrate the ability to utilize
information and solve problems including implementing strategies, accurately performing procedures and
providing solutions.
Proficient The student’s work demonstrates an understanding of basic concepts in mathematics and connections
among central ideas. The student’s responses demonstrate basic ability to solve problems, including
performing procedures and providing solutions. The student’s work may contain minor errors.
Partially Proficient The student’s work demonstrates an incomplete understanding of basic concepts in mathematics and inconsistent connections among central ideas. The student’s responses demonstrate limited ability to solve
problems, including performing procedures and providing solutions. Problem solving strategies may be
flawed and procedures preformed inaccurately.
Substantially Below Proficient The student’s work demonstrates limited understanding of basic concepts in mathematics and inaccurate
connections among central ideas. The student’s responses demonstrate minimal ability to solve problems.
Problem solving strategies may be flawed or inappropriate and there may be many omissions.
Appendix A—Achievement Level Definitions 16 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Science ~ Grade 5
Science achievement level definitions describe the quality of a student’s responses on state-level alternate
assessments in relation to the alternate science standards. These definitions serve as the foundation for
achievement level definitions for the alternate grade level expectations in science.
Skills associated with these achievement level definitions may include some of the following:
• identifying night and day
• identifying the Sun
• identifying the Earth’s Moon
• identifying the position of the sun at
different times
• identifying or drawing different phases
of the Moon
• identifying weather through observation
• identifying different forms that water
can take in the weather
• matching weather to the effects it can
have on the Earth’s surface • indentifying pictures or descriptions
of given animals and plants
• indentifying plants and animals, and
components of the environments that
animals depend on for food and shelter
Appropriate performance at grade five is clarified by the level of complexity within the standard.
Proficient with Distinction The student’s work demonstrates an understanding of essential concepts in science, including the
ability to make connections among central ideas. The student’s response demonstrates the ability
to utilize information and solve problems and explain central concepts with clarity and accuracy.
Proficient The student’s work demonstrates a general understanding of essential concepts in science and connections among central ideas. The student’s response demonstrates the ability to utilize
information and solve problems and explain central concepts. Student work may contain minor
errors.
Partially Proficient
The student’s work demonstrates incomplete understanding of essential concepts in science and
inconsistent connections among central ideas. The student’s response demonstrates some ability
to utilize information and solve problem. The quality of the responses is inconsistent. Explanation
of concepts may be incomplete or unclear.
Substantially Below Proficient The student’s work demonstrates limited understanding of essential concepts in science and infrequent or inaccurate connections among central ideas. The student’s responses demonstrate
minimal ability to solve problems. Explanations are illogical, incomplete, or missing. There are
many inaccuracies.
Appendix A—Achievement Level Definitions 17 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Science ~ Grade 8
Science achievement level definitions describe the quality of a student’s responses on state-level alternate
assessments in relation to the alternate science standards. These definitions serve as the foundation for
achievement level definitions for the alternate grade level expectations in science.
Skills associated with these achievement level definitions may include some of the following:
• identifying or demonstrating ways
objects can move • identifying that an object’s motion can
be changed by pushing or pulling
• identifying or describing wave motions,
earthquakes, vibrations, and/or water
waves • identifying human body parts
• matching animals and/or plants to their
parts
• identifying parts that allow living
things to meet basic needs • identifying that some living things are
made of one cell and some are made
of many cells
• identifying parents and offspring
• demonstrating an understanding of
life cycles, and/or identifying similar
and different characteristics of
offspring and parents
Appropriate performance at grade eight is clarified by the level of complexity within the standard.
Proficient with Distinction The student’s work demonstrates an understanding of essential concepts in science, including the
ability to make connections among central ideas. The student’s response demonstrates the ability
to utilize information and solve problems and explain central concepts with clarity and accuracy.
Proficient The student’s work demonstrates a general understanding of essential concepts in science and connections among central ideas. The student’s response demonstrates the ability to utilize
information and solve problems and explain central concepts. Student work may contain minor
errors.
Partially Proficient The student’s work demonstrates incomplete understanding of essential concepts in science and
inconsistent connections among central ideas. The student’s response demonstrates some ability
to utilize information and solve problem. The quality of the responses is inconsistent. Explanation
of concepts may be incomplete or unclear.
Substantially Below Proficient The student’s work demonstrates limited understanding of essential concepts in science and infrequent or inaccurate connections among central ideas. The student’s responses demonstrate
minimal ability to solve problems. Explanations are illogical, incomplete, or missing. There are
many inaccuracies.
Appendix A—Achievement Level Definitions 18 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Science ~ High School
Science achievement level definitions describe the quality of a student’s responses on state-level alternate
assessments in relation to the alternate science standards. These definitions serve as the foundation for
achievement level definitions for the alternate grade level expectations in science.
Skills associated with these achievement level definitions may include some of the following:
• sorting objects into groups using
physical properties • describing physical properties of objects
and materials
• using observable characteristics to
describe physical changes
• identifying chemical and physical changes
• identifying organisms that are similar
and different based on external features
• describing how plants and/or animals
look
• describing ways in which the needs of
a plant and/or animal are met by its
environment
• sorting living things based on external
features
• matching organisms to the
environment in which they live
• identifying organisms that once lived
on Earth but no longer exist • identifying examples of fossils and/or
explaining how fossils are used to
help us understand the past
Appropriate performance at the high school level is clarified by the level of complexity within the
standard.
Proficient with Distinction
The student’s work demonstrates an understanding of essential concepts in science, including the
ability to make connections among central ideas. The student’s response demonstrates the ability
to utilize information and solve problems and explain central concepts with clarity and accuracy.
Proficient
The student’s work demonstrates a general understanding of essential concepts in science and
connections among central ideas. The student’s response demonstrates the ability to utilize
information and solve problems and explain central concepts. Student work may contain minor
errors.
Partially Proficient The student’s work demonstrates incomplete understanding of essential concepts in science and
inconsistent connections among central ideas. The student’s response demonstrates some ability
to utilize information and solve problem. The quality of the responses is inconsistent. Explanation
of concepts may be incomplete or unclear.
Substantially Below Proficient The student’s work demonstrates limited understanding of essential concepts in science and
infrequent or inaccurate connections among central ideas. The student’s responses demonstrate
minimal ability to solve problems. Explanations are illogical, incomplete, or missing. There are
many inaccuracies.
Appendix A—Achievement Level Definitions 19 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Writing ~ Grade 4
Writing achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate writing standards. These definitions also serve
as the foundation for achievement level definitions for the alternate grade level expectations in
writing.
Skills associated with these achievement level definitions may include some of the following:
• identifying pictures or symbols to relate an experience, event, or idea
• composing responses
• using pictures to create an understandable story line with a beginning and end
• using pictures to identify and/or create characters
Appropriate performance at grade four is clarified by the level of complexity within the standard.
Proficient with Distinction The student’s performance demonstrates understanding of essential concepts in writing. The student’s work demonstrates the ability to compose a response that is well-organized, accurate
and focused.
Proficient The student’s performance demonstrates an understanding of basic concepts in writing. The student’s work demonstrates the ability to compose a response that is organized, accurate and
focused. Some errors may occur but do not interfere with meaning.
Partially Proficient The student’s performance demonstrates an incomplete understanding of basic concepts in writing. The student’s work demonstrates the ability to compose a response that may be
inconsistent and/or limited in its organization, accuracy and/or focus. Some errors may occur that
interfere with meaning.
Substantially Below Proficient The student’s performance demonstrates limited understanding of basic concepts in writing. The student’s work demonstrates the ability to compose a response that is unorganized, inaccurate
and unfocused. Frequent errors may occur that interfere with meaning.
Appendix A—Achievement Level Definitions 20 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Writing ~ Grade 7
Writing achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate writing standards. These definitions also serve
as the foundation for achievement level definitions for the alternate grade level expectations in
writing.
Skills associated with these achievement level definitions may include some of the following:
• composing and sharing related
responses to convey needs
• representing facts through pictures
• using pictures to illustrate details or
information related to topic
• sorting or grouping facts and/or
ideas within a given category
• using pictures and/or words to create
meaning
• including details or information
relevant to topic
• using a given organizational
structure for grouping facts and/or
ideas
• using sufficient details or pictures
to illustrate facts
• using basic transition words
• providing a concluding sentence
Appropriate performance at grade seven is clarified by the level of complexity within the
standard.
Proficient with Distinction The student’s performance demonstrates understanding of essential concepts in writing. The student’s work demonstrates the ability to compose a response that is well-organized, accurate
and focused.
Proficient The student’s performance demonstrates an understanding of basic concepts in writing. The student’s work demonstrates the ability to compose a response that is organized, accurate and
focused. Some errors may occur but do not interfere with meaning.
Partially Proficient The student’s performance demonstrates an incomplete understanding of basic concepts in writing. The student’s work demonstrates the ability to compose a response that may be
inconsistent and/or limited in its organization, accuracy and/or focus. Some errors may occur that
interfere with meaning.
Substantially Below Proficient The student’s performance demonstrates limited understanding of basic concepts in writing. The student’s work demonstrates the ability to compose a response that is unorganized, inaccurate
and unfocused. Frequent errors may occur that interfere with meaning.
Appendix A—Achievement Level Definitions 21 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Writing ~ High School
Writing achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate writing standards. These definitions also serve
as the foundation for achievement level definitions for the alternate grade level expectations in
writing.
Skills associated with these achievement level definitions may include some of the following:
• using phonemic awareness and letter
knowledge to represent initial or
final consonant sounds,
• writing recognizable phrases or short
sentences to show understanding of
text
• using prior knowledge or references
to text to respond to questions or
when reading
• using a beginning and an ending to
organize ideas
• applying basic capitalization and
punctuation rules
• correctly spelling high frequency
words
• writing a variety of simple
sentences
• recognizing or applying English
spelling rules
• stating and maintaining focus
when responding to questions
Appropriate performance at the high school level is clarified by the level of complexity within
the standard.
Proficient with Distinction The student’s performance demonstrates understanding of essential concepts in writing. The student’s work demonstrates the ability to compose a response that is well-organized, accurate
and focused.
Proficient The student’s performance demonstrates an understanding of basic concepts in writing. The student’s work demonstrates the ability to compose a response that is organized, accurate and
focused. Some errors may occur but do not interfere with meaning.
Partially Proficient The student’s performance demonstrates an incomplete understanding of basic concepts in writing. The student’s work demonstrates the ability to compose a response that may be
inconsistent and/or limited in its organization, accuracy and/or focus. Some errors may occur that
interfere with meaning.
Substantially Below Proficient The student’s performance demonstrates limited understanding of basic concepts in writing. The student’s work demonstrates the ability to compose a response that is unorganized, inaccurate
and unfocused. Frequent errors may occur that interfere with meaning.
Appendix A—Achievement Level Definitions 22 2010 Maine PAAP Standard Setting Report
Appendix B—AGENDAS
Appendix B—Agendas 1 2010 Maine PAAP Standard Setting Report
MAINE ALTERNATE ASSESSMENT STANDARD SETTING
June 28 & 29, 2010
AGENDA
Day 1: Monday, June 28th
1st
Grade
All panelists together 8:00 – 8:30 Registration & continental breakfast
8:30 – 8:45 Welcome from Maine’s Department of Education (MDOE)
Introduction of MDOE staff
Introduction of Measured Progress staff 8:45 – 10:00 Overview of the Maine Alternate Assessment
Overview of Standard Setting Process
Panelists break into their respective workgroup rooms 10:00 – 10:15 Break; panelists move to their grade level/content area workgroup rooms 10:15 – 12:00 Introductions of facilitator and panelists
Review of PAAP materials (AGLEs and student samples)
Review and discuss Achievement Level Descriptors
Training Evaluation
All panelists together 12:00 – 12:45 Lunch
Panelists return to their respective workgroup room 12:45 – 4:00 Rounds 1, 2 & 3
Procedural Evaluation Break as needed (approximately 2:00 pm or between rounds)
Day 2: Tuesday, June 29
th
2nd
Grade
All panelists together
7:30 – 8:00 Continental breakfast
Panelists break into their respective workgroup rooms 8:00 – 8:45 Review process, answer questions and check for understanding
8:45 – 12:00 Review of PAAP materials (AGLEs and student samples)
Review and discuss Achievement Level Descriptors
Round 1
Begin Round 2
All panelists together 12:00 – 12:45 Lunch
Panelists return to their respective workgroup room 12:45 – 3:30 Continue with Rounds 2 & 3
Break as needed (approximately 2:00 pm or between rounds)
All panelists together 3:30 – 4:00 Cross grade panel
Final Evaluation
Appendix B—Agendas 3 2010 Maine PAAP Standard Setting Report
MAINE ALTERNATE ASSESSMENT STANDARD SETTING
June 30, 2010
AGENDA
7:30 – 8:00 Registration/room assignments & continental breakfast
Panelists break into their respective workgroup rooms
8:00 – 8:45 Introductions of facilitator and panelists
Review process, answer questions and check for understanding
8:45 – 12:00 Review of PAAP materials (AGLEs and student samples)
Review and discuss Achievement Level Descriptors
Round 1
Begin Round 2
All panelists together 12:00 – 12:45 Lunch
Panelists return to their respective workgroup room
12:45 – 4:00 Continue with Rounds 2 & 3
Final evaluation
Break as needed (approximately 2:00 pm or between rounds)
Appendix B—Agendas 4 2010 Maine PAAP Standard Setting Report
Appendix C—NONDISCLOSURE FORM
Appendix C—Nondisclosure Form 1 2010 Maine PAAP Standard Setting Report
Maine Personalized Alternate Assessment Portfolio
Nondisclosure Agreement for Standard Setting Panelists
The student work associated with the Maine Personalized Alternate Assessment Portfolio (PAAP) is confidential material. As such, the student portfolio may not be copied, shared, or discussed for any reason other than to score the student work. It is the policy of the Maine Department of Education that student portfolios be treated as private and secure material.
The undersigned is a PAAP In-State Standard Setting participant authorized to view PAAP material and hereby agrees to be bound to the terms of this agreement restricting the disclosure of said materials.
Name (printed)
Name (signature)
Date
Appendix C—Nondisclosure Form 3 2010 Maine PAAP Standard Setting Report
Appendix D—SAMPLE RATING FORM
Appendix D—Sample Rating Form 1 2010 Maine PAAP Standard Setting Report
Round: ID Number:
Maine PAAP Reading, Grade 2
SBP PP P PWD
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Appendix D—Sample Rating Form 3 2010 Maine PAAP Standard Setting Report
Appendix E—SAMPLE EVALUATION
Appendix E—Sample Evaluation 1 2010 Maine PAAP Standard Setting Report
Str
ongly
Dis
agre
e
Dis
agre
e
Undec
ided
Agre
e
Str
ongly
Agre
e
Content Area:
Grade:
Standard Setting Training Evaluation
The purpose of this evaluation form is to obtain your feedback about the training you have received.
Please complete the information below. Do not put your name on the form. We want your feedback
to be anonymous.
Please mark the appropriate box for each statement.
I understand the goals of the standard setting meeting. □ □ □ □ □
I understand the procedures we are using to set standards. □ □ □ □ □
I understand how to use the standard setting materials. □ □ □ □ □
I understand the differences between the achievement levels. □ □ □ □ □
I understand how to make the cut score judgment. □ □ □ □ □
I know what tasks to expect for the remainder of the meeting. □ □ □ □ □
I am confident in my understanding of the standard setting task. □ □ □ □ □
Please indicate any areas in which you would like more information before you continue.
Please indicate any questions you may have about the remainder of the standard setting meeting.
Appendix E—Sample Evaluation 3 2010 Maine PAAP Standard Setting Report
Not
at a
ll
infl
uen
tial
Str
ongly
Dis
agre
e
Dis
agre
e
Undec
ided
Agre
e
Extr
emel
y
infl
uen
tial
Str
ongly
Agre
e
Content Area:
Grade:
Standard Setting Procedural Evaluation
Please mark the appropriate box for each statement.
I understood how to make the cut score judgments. □ □ □ □ □
I understood how to use the materials provided. □ □ □ □ □
I understood how to record my judgments. □ □ □ □ □
I think the procedures make sense. □ □ □ □ □
I am sufficiently familiar with the assessment. □ □ □ □ □
I understand the differences between the achievement levels. □ □ □ □ □
Please indicate any questions you may have about the remainder of the standard setting meeting.
Please rate the influence of the following when setting standards.
The achievement level descriptors. □ □ □ □ □
The state content standards. □ □ □ □ □
My perception of the difficulty level of the assessment. □ □ □ □ □
The student responses. □ □ □ □ □
My experience working with students. □ □ □ □ □
What materials, information, or procedures were most influential in your placement of the cut scores?
Why?
Appendix E—Sample Evaluation 4 2010 Maine PAAP Standard Setting Report
Str
ongly
Dis
agre
e
Dis
agre
e
Undec
ided
Agre
e
Str
ongly
Agre
e
Content Area:
Grade:
Standard Setting Final Evaluation
Please complete the information below. Your feedback will provide a basis for evaluating the training,
methods, and materials. Do not put your name on the form. We want your feedback to be anonymous.
Gender: Male □ Female □ Race/ethnicity: White □ Black □ Hispanic □ Asian □ Pacific Islander □ American Indian □ Years of experience in education: 0-5 □ 5-10 □ 10-15 □ More than 15 □ Area of Expertise (Check all that apply): Students with Disabilities □
Students with Limited English Proficiency □ Economically Disadvantaged Students □ Gifted and Talented Students □ General Education □
Please mark the appropriate box for each statement.
I understood the goals of the standard setting meeting. □ □ □ □ □
I understood the procedures we used to set standards. □ □ □ □ □
The facilitator helped me understand the process. □ □ □ □ □
The materials contained the information needed to set standards. □ □ □ □ □
I understood how to use the materials provided. □ □ □ □ □
The achievement level descriptors were clear. □ □ □ □ □
I understood how to make the cut score judgments. □ □ □ □ □
I understood how to use the feedback provided after each round. □ □ □ □ □
I understood how to use the impact data. □ □ □ □ □
I understood how the cut scores were calculated. □ □ □ □ □
The facilitator was able to get answers to my questions. □ □ □ □ □
Sufficient time was allotted for training on the standard setting tasks. □ □ □ □ □
Sufficient time was allotted to complete the standard setting tasks. □ □ □ □ □
Appendix E—Sample Evaluation 5 2010 Maine PAAP Standard Setting Report
Not
at a
ll
infl
uen
tial
Not
at a
ll
use
ful
Extr
emel
y
infl
uen
tial
Extr
emel
y
use
ful
Content Area:
Grade:
The facilitator helped the standard setting process run smoothly. □ □ □ □ □
Please rate the usefulness of each of the following:
The opening session. □ □ □ □ □
The small group activities. □ □ □ □ □
Becoming familiar with the assessment. □ □ □ □ □
Articulating the differences between the achievement levels. □ □ □ □ □
Discussions with other participants.
Please rate the influence of the following when setting standards.
□ □ □ □ □
The achievement level descriptors. □ □ □ □ □
My expectations of students. □ □ □ □ □
The difficulty of the test materials. □ □ □ □ □
The student responses. □ □ □ □ □
My experience in the field. □ □ □ □ □
Discussions with other participants. □ □ □ □ □
Cut scores of other participants. □ □ □ □ □
Impact data. □ □ □ □ □
Please provide any additional comments about the standard setting process or suggestions as to how the
training and process could be improved.
Appendix E—Sample Evaluation 6 2010 Maine PAAP Standard Setting Report
APPENDIX F—OPENING SESSION POWERPOINT
Appendix F—Opening Session PowerPoint 1 2010 Maine PAAP Standard Setting Report
Slide 1
Slide 2
Maine’s Personalized
Alternate Assessment
Portfolio (PAAP)
Standard Setting
Introductions
Maine’s Department of Education
- Susan Fossett, PAAP and Accommodations
Coordinator
- Peter Bernard, PAAP Assistant
Measured Progress (Program Mgt.)
- Susan Izard, Division Director
- Sharon Houle, PAAP Program Manager
- Stephanie Arroyo, PAAP Specialist
- Michelle Couture, PAAP Program Assistant
Slide 3
Data
Introductions (cont.)
- Liz Burton, Psychometrician
- Jennifer Dunn, Psychometrician
- Kevin Froton, Data Processing
Slide 4
Introductions (cont.)
Facilitators (Monday and Tuesday)
Grade 2/3 Reading: Amanda Breitmaier
Grade 2/3 Mathematics: Alicia Cuttle
Grade 4/5 Reading: Jaime Alford
Grade 4/5 Mathematics: Stephanie Arroyo
Grade 6/7 Reading: Tim Greenlaw
Grade 6/7 Mathematics: Susan Izard
Grade 10/11 Reading: Jake Goldsmith
Grade 10/11 Mathematics: Kristen Cole
Slide 5 Introductions (cont.)
Facilitators (Wednesday):
Science
Grade 5: Alicia Cuttle
Grade 8: Stephanie Arroyo
Grade 11: Kristen Cole
Writing
Grade 4: Amanda Breitmaier
Grade 7: Susan Izard
Grade 11: Jake Goldsmith
Slide 6
Agenda
8:30 – 10:00 am General Session
10:00 am – 12:00 pm Breakout Rooms
12:00 – 12:45 pm Lunch
12:45 – 4:00 pm Breakout Rooms
Breaks as needed in Breakout Rooms
Slide 7 The PAAP is
What is the PAAP?
- designed for students with significant cognitive
disabilities who meet participation criteria;
- administered during the “teaching year” at the
same grade levels as students in general
education are assessed in October via NECAP;
- a collection of student work;
- assessed during the PAAP assessment window
December 1st through April 30th;
Slide 8
The PAAP is
What is the PAAP?
- tasks that teachers download to assess
students at an instructionally appropriate time;
- tasks selected by teachers that best fit their
students (using the test blueprint as a guide);
- scored using the PAAP Scoring Rubric to
obtain student achievement levels which are
then used to determine reportable scores; and
- reflective of input from an instructional team.
Slide 9 2009-10 Test Blueprint
Slide 10
Visual Guide to PAAP Requirements
Slide 11 Steps to Administer a PAAP
STEP 1: The teacher/IEP team determines a student is
eligible to participate for each content area.
STEP 2: The teacher reviews the Alternate Grade Level Expectations (AGLE) document to determine which
AGLE/Indicator and Level of Complexity (LoC) is
appropriate from the eight LoCs.
STEP 3: The teacher selects/downloads tasks from the
PAAP Task Bank (including graphics/passages). STEP
4: The teacher administers the tasks to the student. STEP
5: The teacher corrects/grades the student work and
provides details regarding the level of assistance
provided to the student.
Slide 12
Slide 13
Slide 14
Slide 15
What does a Task look like?
1. Entry Slip
2. Task Description
3. Student Work (template)
and graphics
4. Task Summary
Slide 16
Slide 17
Slide 18
The Entry Slip • Teacher fills in Student
Name and Grade.
• The Content Area, AGLE, Level of Complexity (LoC), and the AGLE/Indicator is pre-populated.
• By design, text provided in gray shaded cells is information for the teacher and is pre- populated.
• Areas with white or no shading indicates the teacher should fill in the missing information.
• LoC 1 for ELA
Template Pages
Slide 19
Slide 20 Task Description Page
• The Task Description page repeats the Content Area, AGLE/Indicator, LoC information.
• The Task Title, AGLE Page #, Task # has been added to the top identifying information.
• Prior Knowledge, Description of Task, Directions for Task Administration, and Responses Expected are provided to ease teacher administration.
Slide 21 Work Template Page
• LoC 1 Work
are always a chart
similar to this one.
(Reading & Writing)
always includes
color images.
Option 2 – Always Available at LoC 1 • Option 2 allows for fewer
of the item sets to be used
multiple times when
necessary to match the level of student knowledge.
• The items must be selected from those already provided
in the Task.
• Option 2 allows for the use of
a combination of selected sets that total six items (e.g.,
two sets each used on 3
different occasions…or
they can use the same set
on 6 different occasions).
Slide 22
How the Item 1 is Presented to the
Student
Slide 23
Slide 24 Liam 12/11/10
Option 2
sleeping
smiling
sleeping
smiling
sleeping
smiling
eating
person laughing
eating
person laughing
eating
person laughing
sleeping
sleeping
sleeping
eating
eating
eating
sleCeping
smilXing
slCeeping
eatCing
Ceating
laughXing
Liam 12/11/10
67
The Completed Work Template 1. Teacher fills in Liam 12/11/10
student name, date,
and student
responses. C
X
2. Teacher corrects C
C student responses
C using the Responses
Expected from X
Student section on
the Task Description.
response
Hand-over-Hand
Slide 25
Slide 26 Scoring Level of Accuracy
1. Determine the Level of Accuracy based on
the corrected student work and Data Key. – Point values are predetermined. NO PARTIAL POINTS
unless noted.
– % are pre-calculated.
Task 1 Summary
Slide 27 Scoring Level of Assistance
Unscorable:
●
● Altering Items/Tasks (task no longer connects to
the AGLE.)
Level of Assistance Score of 1:
● Modeling
● Demonstrating a response similar to the desired
X
Liam required modeling of Item 1 to understand what was being
asked of her. He proceeded without further assistance.
Scoring Level of Assistance
2. Determine Level of Assistance based
Administration Handbook.
.
Slide 28 Scoring Level of Assistance
Level of Assistance Score of 2:
• Use of Option 2 (LoC 1 only) to use fewer of the
item sets multiple times to match student
knowledge
• Limiting a student’s response (outside of LoC 1 at
Option 2) by removing one response option
• Use of clarifying questions to stimulate student
thought to the specific task without providing clues
to specific answers
Slide 29 Scoring Level of Assistance
Level of Assistance Score of 3:
• Independent
• Providing encouragement
• Completing Task by using
augmentative/alternative means of
communication
• Repeating directions
• Reacting to a student
• Rereading a passage (except for required
reading)
• Reminding a student to stay focused
Slide 30
upon criteria outlined in the PAAP
Scoring Level of Assistance
3.The bottom of the Task Summary is a reminder to
the teacher if the Student Work, Level of Accuracy,
and Level of Assistance boxes are not completed
and submitted together by Task, the Task becomes
UNSCORABLE.
Slide 31
Slide 32 Tasks 2 and 3
• Usually at LoC 1, Tasks 2 and/or 3 are a repeat
of Task 1 with different pictures, with the same
concept being assessed.
• As you move to higher LoCs, Tasks 2 and 3
build off Task 1, getting progressively more
difficult/complex. The skills may be different
depending on what is required of the LoC.
• Reading and Writing require 3 Tasks per AGLE
Entry.
• Mathematics and Science require 2 Tasks per
AGLE Entry.
Slide 33
What does the PAAP Assess?
√ Level of Complexity
√ Level of Accuracy
√ Level of Assistance
A final reminder…
☺ PAAPs were scored by two scorers,
sometimes three.
☺ Do not attempt to rescore the samples
your reviewing!
☺ You may disagree about the order of the
PAAPs; that’s fine.
☺ You need to stay focused on the task at
hand!
Slide 34 Who scored the PAAPs?
● Measured Progress scorers scored 2,024
PAAPs in early May using the PAAP Task
Scoring Rubric. Each PAAP was scored twice
and sometimes a third time if scores between
Scorer #1 and #2 were not an exact match.
● Measured Progress provided training.
● All scorers passed a qualifying test.
● MDOE staff were present at scoring and
available to answer questions.
Slide 35 PAAP Task
Scoring Rubric
Slide 36
Slide 37
Slide 38
Now the process…
Today’s Training
In today’s session we will cover:
1. an overview of standard setting;
2. details of the Body of Work process as it
will be implemented for the PAAP; and
3. your role in this process.
Note:
This session is intended to be an overview.
Your facilitator will give you more details and will guide you through the process step by step.
Slide 39 Logistical Overview Monday/Tuesday
Reading 2/3
Reading 4/5
Reading 6/7
Reading 10/11
Math 2/3
Math 4/5
Math 6/7
Math 10/11
Wednesday
Science 5
Science 8
Science 11
Writing 4
Writing 7
Writing 11
Proficient
Slide 40
Overview of Standard Setting
Slide 41 Content
Standards
vs.
Achievement
Standards
Content standards (AGLE) = “What”
Describe the knowledge and skills students are expected to
demonstrate by content area and grade span
Achievement standards (e.g., Proficient) =
“How well”
Describe attributes of student performance based on Achievement Level Descriptors
Slide 42
What is Your Job?
To recommend cut scores for each of the
achievement levels that will be used to report
PAAP results:
Substantially Below Proficient
Partially Proficient
Proficient with Distinction
Based on Achievement Level
Descriptors, you will recommend cut
scores…
Cut score Cut score Cut score needed needed needed
Substantially Partially Proficient Proficient Below Proficient with Proficient Distinction
Achievement Continuum
Slide 43 We are trying to determine What knowledge, skills, and abilities (KSAs)
need to be demonstrated to be classified in each achievement level?
How much is enough?
What portfolio evidence corresponds to:
Substantially Below Proficient
Partially Proficient
Proficient
Proficient with Distinction
Slide 44
Achievement Continuum
Substantially
Below Proficient
Partially Proficient
Proficient Proficient
with Distinction
Slide 45
Slide 46
General Phases of Standard Setting
Data-collection
Policy-making/Decision-making
Slide 47
Final Recommendations
Your recommendations will be reviewed and
presented to the policy makers, who are
responsible for final adoption of the cut
scores.
The recommendations may be accepted or
modified by the Maine Department of Education including the Commissioner of
Education and the Technical Advisory Committee.
Slide 48
Overview of Standard Setting
Method
Slide 49 Cut Score Recommendations
Provide data to establish the
following cut scores:
Substantially Below Proficient
Partially Proficient
Proficient
Proficient with Distinction
Cut Score
Cut Score
Cut Score
Slide 50 How: The Body of Work Method
Examine student work and make a judgment
regarding the achievement level to which the
student work most closely corresponds.
Student Work Samples (portfolios or PAAPs) 30 to 40 student PAAPs
Your job is to classify each portfolio into the
achievement level in which you feel it
belongs.
Slide 51 Why the Body of Work?
Allows panelists to use samples of actual student
work to make their determinations
Is especially useful for assessments that consist
primarily or entirely of performance-based
items
Has been used successfully for setting standards
on similar assessments in the past
Has resulted in defensible cut points
Slide 52 General Process
Classify each portfolio into
one of 4 achievement
levels based on the
following:
Achievement Level Descriptors
KSAs measured by the portfolios
How the students performed on the portfolios
Slide 53
Before you start classifying portfolios….
You will need to become familiar with:
√ Alternate Grade Level Expectations (AGLEs)
√ Achievement Level Descriptors
What each level means
Identify the knowledge, skills and abilities
necessary to be classified in each level
√ Student PAAPs/portfolios
Understand the knowledge, skills and
abilities demonstrated in the work samples
Slide 54 Achievement Level Descriptors Individual review of Achievement Level Descriptors
Group discussion of what performance in each
achievement level looks like
Create bulleted lists of
The knowledge, skills, and abilities a student must demonstrate to be classified in each achievement level
The knowledge, skills, and abilities that distinguish one achievement level from another
You must reach consensus as a group about
the KSAs that define student performance at each achievement level.
Slide 55
Student Portfolios You will classify 30 to 40 student portfolios.
The portfolios cover the range of possible total
scores and are presented in order from lowest
(e.g., Sample #1) to highest (e.g., Sample #35)
total raw score.
Each portfolio has been selected because it
shows typical types of evidence submitted for
students who received a given total score.
Slide 56 Your Task
Think about a student who demonstrates the
KSAs for each level.
Classify each portfolio into the level you feel it
belongs:
Substantially Below Proficient
Partially Proficient
Proficient
Proficient with Distinction
Slide 57
Slide 58
Slide 59
Please Note: You may disagree about the order of the portfolios; that’s fine.
You will categorize the portfolios as you see fit, whether your ratings agree with the order or not.
However, it is not your job to rescore the portfolios; you need to stay focused on the task at hand.
Slide 60
Round 1 Working Individually:
Review each portfolio
Focus on the knowledge, skills and abilities being
demonstrated in the portfolio
Determine which Achievement Level Descriptor best
matches the knowledge, skills and abilities
demonstrated in the portfolio
Classify the portfolio into the appropriate
achievement level
Complete the rating form
of the discussions.
Slide 61 Round 2 Working as a Group:
Discuss your portfolio classifications in relation to
The average round 1 results
The other panelists
The knowledge, skills and abilities
Working Individually:
Determine which Achievement Level Descriptor best matches the knowledge, skills and abilities demonstrated in the portfolio
Classify the portfolio into the appropriate achievement level
Complete the rating form
Slide 62
Round 3 Working as a Group:
Discuss your portfolio classifications in relation to
The round 2 results & impact data
The other panelists
The knowledge, skills and abilities
Working Individually:
Determine which Achievement Level Descriptor best matches the knowledge, skills and abilities
demonstrated in the portfolio
Classify the portfolio into the appropriate achievement level
Complete the rating form
Slide 63 A few reminders
It is not necessary for panelists to reach
consensus as to how the portfolios should be
categorized.
You should be open-minded when listening to
your colleagues’ rationales for their ratings.
You may or may not change your mind as a result
We want each panelist to use his or her own best
judgment in each round of rating.
Slide 64 Cross-Grade Policy Forum
(Tuesday afternoon)
After all groups have completed Round 3 for the
second grade level, the groups for each
content area (reading and mathematics) will
meet together to look at results across grades
and provide feedback.
Slide 65
Evaluation
At several different points in the process, we
will ask you to complete an anonymous
evaluation of the standard setting procedures.
Your honest feedback is important for
improving future standard settings, and for
evaluating the results of this one.
Slide 66
Questions about the Body of Work
Method?
Slide 67
Before you break into groups…
Slide 68 Top 10 Misconceptions About
Standard Setting
10. Standard setting is a great opportunity to
review and revise the PAAP.
9. Standard setting is the same thing as scoring.
8. This is a good time to discuss PAAP
administration policy.
7. This is a good time to revise the content
standards.
6. This is a good time to revise the PAAP
Achievement Level Descriptors.
Slide 69
Top 10 Misconceptions About Standard
Setting 5. This is a good time to discuss effective
teaching strategies.
4. Only scholars and researchers are qualified
to do this work.
3. Only educators are qualified to do this work.
2. The process is rigged.
1. Disagreement is bad.
Slide 70
What Next?
☺ Some meeting logistics
☺ After this session, you will break into grade
groups and complete the standard setting
process! – First grade/content
• Review the portfolios
• Discuss the Achievement Level Descriptors
• Round 1, 2, 3
– Repeat for second grade/content
– Cross grade policy forum
– Evaluation
☺ Some lucky folks will – Repeat for third grade/content
Slide 71 Room Assignments
Please refer to your hotel map with
room assignments in your folder.
Appendix G—FACILITATORS’ SCRIPTS
Appendix G—Facilitators’ Scripts 1 2010 Maine PAAP Standard Setting Report
GENERAL INSTRUCTIONS FOR GROUP FACILITATORS MAINE PAAP STANDARD SETTING
READING AND MATHEMATICS
JUNE 28-30, 2010
Overview
The Reading and Mathematics groups will each be setting standards for two grade levels. The
panels will complete the standard setting activities for the first grade level – discussing the
Achievement Level Descriptors and completing the three rounds of ratings – then will repeat the
entire process for the second grade level. For the first grade level, the panelists will complete two
evaluation forms: a training evaluation before starting round 1 and a procedural evaluation after
round 3. For the second grade level, the panelists will not need to fill out either of these
evaluations; instead, the panelists will complete the final evaluation after the Cross Grade Policy
Forum, which occurs at the very end of the process.
Introductions
1) Welcome group, introduce yourself (name, affiliation, a little selected background
information).
2) Have each participant introduce him/herself.
3) Ask each participant to sign a nondisclosure form. Do not proceed until a signed
nondisclosure form has been collected from each participant.
Complete Standard Setting Activities for First Grade Level Review Materials
1) Have the panelists take a few minutes to briefly look through the AGLEs 2) Have the panelists briefly review about every fifth portfolio, noting the increasing level of
performance.
3) When they are done, allow a minute or two for comments or questions.
4) Familiarize the panelists with the rating sheet and explain how to complete it:
a. Place one (and only one) “X” in each row
b. They can place the X’s at the low or high end of the box, but they must clearly be
within one box: no straddling!
Appendix G—Facilitators’ Scripts 3 2010 Maine PAAP Standard Setting Report
Discuss Achievement Level Descriptors
Overview: In order to establish a thorough understanding of the expected performance of students
on the test, panelists must have a clear understanding of:
1) The definitions of the four achievement levels, and
2) what the key characteristics are that distinguish students in adjacent achievement level
categories.
The purpose of this activity is for the panelists to come to consensus about what characterizes
students in each of the four achievement level categories. This activity is critical since the ratings
panelists will be making in Rounds 1 through 3 will be based on these understandings.
Activities:
1. Introduce task. In this activity they will:
a. Individually review the Achievement Level Descriptors that describe the main
characteristics that define students in each achievement level category; and
b. discuss Descriptors as a group.
2. Have panelists individually review the Achievement Level Descriptors for each level. They
can make notes if they like. The goal here is for the panelists to come to a common
understanding of what it means to be in each achievement level. It is not unusual for
panelists to disagree with the Descriptors they will see; almost certainly there will be some
panelists who will want to change them. However, the task at hand is for panelists to have a
common understanding of what knowledge, skills, and abilities are described by each
Achievement Level Descriptor.
3. After individually reviewing the Descriptors, have the panelists discuss each achievement
level as a group, starting with Partially Proficient. The panelists will discuss the
characteristics a student must demonstrate in order to be classified in the Partially Proficient
category. Or, put another way, the most important characteristics that distinguish a
Substantially Below Proficient student from a student in the Partially Proficient category. They
will then repeat this process for the Proficient and Proficient with Distinction categories. The
purpose of this step is to have a collegial discussion in which to bring up/clarify any issues
or questions that any individual may have and to reach consensus on an understanding of
the Descriptors.
4. Have the panelists identify the most important characteristics describing students at each
achievement level and record those as bulleted lists on chart paper. These should be posted
on the walls for panelists to refer to as they complete the three rounds of rating.
Training Evaluation (First Grade Level Only)
After completing the discussion of the Achievement Level Descriptors for the first grade level, have
panelists fill out the training evaluation form before proceeding to Round 1. Before you start the
Round 1 activities, scan the completed evaluations to see if there are any problems or concerns that
need to be addressed before proceeding. Return the completed evaluations to the data analysis
room at the next convenient opportunity. It is not necessary to complete the training evaluation
form for the second grade level.
Appendix G—Facilitators’ Scripts 4 2010 Maine PAAP Standard Setting Report
Round 1 Ratings
Overview of Round 1: The primary purpose of Round 1 is to ask the panelists to make their initial
determination as to which achievement level category each portfolio should be classified into. In
this round, panelists will be working individually, without discussion with their colleagues.
Activities:
1. Make sure panelists have the following materials:
a. Rating form
b. set of portfolios
c. Achievement Level Descriptors
2. Orient panelists to the set of portfolios. Point out that the portfolios are presented in order,
from lowest scoring to highest. Make sure panelists understand that, even though the
portfolios are presented from lowest- to highest-scoring, their own ratings do not need to be
in strictly increasing order.
3. Starting with the first portfolio, the panelists will review each portfolio in turn. As they are
reviewing the portfolios, the panelists should keep in mind the Achievement Level
Descriptors. They should consider the knowledge, skills, and abilities demonstrated by
each and how they relate to the definitions of the achievement levels. The purpose of this
step is for panelists to make their initial determinations as to how the portfolios should be
categorized into the four achievement levels. The panelists are free to make notes on the
portfolios, sort them into piles, use sticky notes, or use whatever system helps them to keep
track of their categorizations.
4. Panelists may want to take notes as they work if there are particular points they would like
to discuss with their colleagues in Round 2. a. Have panelists write their ID and round number on the rating form. The ID number
is on their name tags.
b. Briefly remind them how to fill in the rating form.
c. Answer questions the panelists may have about the work in Round 1.
d. Once everyone understands what they are to do in Round 1, tell them to begin.
5. As panelists complete the task, ask them to carefully inspect their rating forms to ensure
they are filled out properly.
a. The ID and round number must be filled in.
b. Each portfolio must be assigned to one and only one achievement level.
c. Reiterate that although the portfolios are presented in order from lowest- to highest-
scoring, the panelists’ category assignments do not need to be in strictly increasing
order.
6. Facilitators should bring all the completed rating forms together to the data analysis work
room for tabulation. Prior to submitting them, however, using a show of hands, indicate on
a piece of chart paper how many panelists assigned each portfolio to each achievement level
category. This chart will be used for the Round 2 discussions.
Tabulation of Round 1 Results Tabulation of Round 1 results will be completed as quickly as possible after receipt of the rating forms. While the tabulation occurs, the group may take a break.
Appendix G—Facilitators’ Scripts 5 2010 Maine PAAP Standard Setting Report
Round 2 Ratings
Overview of Round 2: In Round 2, the panelists will have an opportunity to discuss their Round 1
placements and to revise their ratings on the basis of that discussion. Prior to beginning the Round
2 discussions, the psychometrician will share the group average cut points based on the Round 1
ratings.
Focusing on any portfolios for which there was disagreement as to how they should be categorized,
the panelists will discuss why they categorized each portfolio as they did, making sure that all
different points of view are included in the discussion.
Once panelists have reviewed and discussed the Round 1 categorizations, the panelists will make
their Round 2 ratings.
Activities:
1. Make sure panelists have the following materials:
a. Rating form
b. set of portfolios
c. Achievement Level Descriptors
2. Have panelists write their ID number and Round 2 on the rating form.
3. Provide an overview of Round 2. Paraphrase the following:
a. As in Round 1, the primary purpose is to categorize each portfolio into the
achievement level category where you believe it belongs.
b. Each panelist needs to base his/her judgments on his/her experience with the
content area, understanding of students, the definition of each achievement level
category, discussions with other panelists, and the knowledge, skills, and abilities
required to answer each item.
4. Show the panelists how the portfolios would be categorized based on the room average
Round 1 cut point placements.
5. Remind panelists that they will be discussing each portfolio with their colleagues, but that
they will be categorizing the portfolios individually. It is not necessary for the panelists to
reach consensus about how to categorize each portfolio.
6. Give panelists an opportunity to ask questions about the task for Round 2.
7. Beginning with the first portfolio for which there is disagreement as to how it should be
categorized, the panelists should begin discussing the categorization of the portfolios
according to the Round 1 ratings. a. Panelists only need to discuss those portfolios for which there was disagreement as
to how they should be categorized.
b. Panelists should be encouraged to listen to their colleagues as well as express their
own points of view.
c. If the panelists hear a logic/rationale/argument that they did not consider and that
they feel is compelling, then they may adjust their ratings to incorporate that
information.
d. On the basis of the discussions and the feedback presented, panelists should make
their round 2 ratings.
Appendix G—Facilitators’ Scripts 6 2010 Maine PAAP Standard Setting Report
e. The group does not have to achieve consensus. If panelists honestly disagree, that is
fine. We are trying to get the best judgment of each panelist. Panelists should not feel
compelled or coerced into making a rating with which they disagree.
Encourage the panelists to use the discussion and feedback to assess how stringent or
lenient a judge they are. If a panelist is categorizing portfolios consistently higher or
lower than the group, he or she may have a different understanding of the Achievement
Level Descriptors than the rest of the group. It is acceptable for panelists to disagree,
but that disagreement should be based on a common understanding of the
Achievement Level Descriptors.
8. When the group has completed their Round 2 ratings, collect the rating forms. When you
collect the rating forms, carefully inspect them to ensure they are filled out properly.
a. The ID and round number must be filled in.
b. Each portfolio must have one (and only one) rating.
9. Facilitators should bring all the completed rating forms together to the data analysis work
room for tabulation. Prior to submitting them, however, using a show of hands, indicate on
a piece of chart paper how many panelists assigned each portfolio to each achievement level
category. This chart will be used for the Round 3 discussions.
Appendix G—Facilitators’ Scripts 7 2010 Maine PAAP Standard Setting Report
Round 3 Ratings
Overview of Round 3: In Round 3, the panelists will have a final opportunity to discuss their
Round 2 placements and to revise their ratings on the basis of that discussion. Prior to beginning
the Round 3 discussions, the psychometrician will share the Round 2 results with the group,
including the group average cut points and impact data, i.e., the approximate percentage of
students who would be classified into each achievement level category based on the room average
cut points from Round 2.
Once panelists have reviewed and discussed the Round 2 categorizations, they will make their final
ratings.
Activities:
1. Make sure panelists have the following materials:
a. Rating form
b. set of portfolios
c. Achievement Level Descriptors
2. Have panelists write their ID number and Round 3 on the rating form.
3. Provide an overview of Round 3. Paraphrase the following:
a. As in Round 2, the primary purpose is to categorize each portfolio into the
achievement level category where you believe it belongs.
b. Each panelist needs to base his/her judgments on his/her experience with the
content area, understanding of students, the definition of each achievement level
category, discussions with other panelists, and the knowledge, skills, and abilities
required to answer each item. 4. Review the feedback information with the panelists.
a. Show the panelists how the portfolios would be categorized based on the room
average Round 2 cut point placements.
b. Go over the impact data, explaining that if the Round 2 ratings were to be used to set
the final cut points, these are the approximate percentages of students who would be
classified into each achievement level category.
5. Remind panelists that they will be discussing each portfolio with their colleagues, but that
they will be categorizing the portfolios individually. It is not necessary for the panelists to
reach consensus about how to categorize each portfolio.
6. Give panelists an opportunity to ask questions about the feedback information or about the
task for Round 3. 7. Beginning with the first portfolio for which there is disagreement as to how it should be
categorized, the panelists should begin discussing the categorization of the portfolios
according to the Round 2 ratings.
a. Panelists should discuss the portfolios for which there was disagreement as to how
they should be categorized, focusing in particular on thoe portfolios around the cuts.
b. Panelists should be encouraged to listen to their colleagues as well as express their
own points of view.
c. If the panelists hear a logic/rationale/argument that they did not consider and that
they feel is compelling, then they may adjust their ratings to incorporate that
information.
Appendix G—Facilitators’ Scripts 8 2010 Maine PAAP Standard Setting Report
d. On the basis of the discussions and the feedback presented, panelists should make
their final ratings.
e. The group does not have to achieve consensus. If panelists honestly disagree, that is
fine. We are trying to get the best judgment of each panelist. Panelists should not feel
compelled or coerced into making a rating with which they disagree.
Encourage the panelists to use the discussion and feedback to assess how stringent or
lenient a judge they are. If a panelist is categorizing portfolios consistently higher or
lower than the group, he or she may have a different understanding of the Achievement
Level Descriptors than the rest of the group. It is acceptable for panelists to disagree,
but that disagreement should be based on a common understanding of the
Achievement Level Descriptors.
8. When the group has completed their final ratings, collect the rating forms. When you collect
the rating forms, carefully inspect them to ensure they are filled out properly.
a. The ID and round number must be filled in.
b. Each portfolio for Round 3 must have one (and only one) rating.
c. Return the completed rating forms to the data analysis work room.
Complete Procedural Evaluation After the panelists have completed the standard setting activities for the first grade level, have them
complete the procedural evaluation. Submit the completed evaluations to the data analysis work
room at the earliest convenient opportunity.
Complete Standard Setting Activities for Second Grade Level
After the panelists have completed the three rounds of ratings and filled in the procedural
evaluation, they will then repeat the standard setting activities (except the training and procedural
evaluations) for the second grade level: discussing the Achievement Level Descriptors and the
three rounds of ratings.
Complete Final Evaluation Form Following the cross grade panel, have panelists fill out the final evaluation form. Emphasize that their honest feedback is important.
Appendix G—Facilitators’ Scripts 9 2010 Maine PAAP Standard Setting Report
GENERAL INSTRUCTIONS FOR GROUP FACILITATORS MAINE PAAP STANDARD SETTING
SCIENCE AND WRITING
JUNE 30, 2010
Introductions
4) Welcome group, introduce yourself (name, affiliation, a little selected background
information).
5) Have each participant introduce him/herself.
6) Ask each participant to sign a nondisclosure form. Do not proceed until a signed
nondisclosure form has been collected from each participant.
Review Materials 5) Have the panelists take a few minutes to briefly look through the AGLEs
6) Have the panelists briefly review about every fifth portfolio, noting the increasing level of
performance.
7) When they are done, allow a minute or two for comments or questions.
Discuss Achievement Level Descriptors
Overview: In order to establish a thorough understanding of the expected performance of students
on the test, panelists must have a clear understanding of:
3) The definitions of the four achievement levels, and
4) what the key characteristics are that distinguish students in adjacent achievement level
categories.
The purpose of this activity is for the panelists to come to consensus about what characterizes
students in each of the four achievement level categories. This activity is critical since the ratings
panelists will be making in Rounds 1 through 3 will be based on these understandings.
Appendix G—Facilitators’ Scripts 10 2010 Maine PAAP Standard Setting Report
Activities:
5. Introduce task. In this activity they will:
c. Individually review the Achievement Level Descriptors that describe the main
characteristics that define students in each achievement level category; and
d. discuss Descriptors as a group.
6. Have panelists individually review the Achievement Level Descriptors for each level. They
can make notes if they like. The goal here is for the panelists to come to a common
understanding of what it means to be in each achievement level. It is not unusual for
panelists to disagree with the Descriptors they will see; almost certainly there will be some
panelists who will want to change them. However, the task at hand is for panelists to have a
common understanding of what knowledge, skills, and abilities are described by each
Achievement Level Descriptor.
7. After individually reviewing the Descriptors, have the panelists discuss each achievement
level as a group, starting with Partially Proficient. The panelists will discuss the
characteristics a student must demonstrate in order to be classified in the Partially Proficient
category. Or, put another way, the most important characteristics that distinguish a
Substantially Below Proficient student from a student in the Partially Proficient category. They
will then repeat this process for the Proficient and Proficient with Distinction categories. The
purpose of this step is to have a collegial discussion in which to bring up/clarify any issues
or questions that any individual may have and to reach consensus on an understanding of
the Descriptors.
8. Have the panelists identify the most important characteristics describing students at each
achievement level and record those as bulleted lists on chart paper. These should be posted
on the walls for panelists to refer to as they complete the three rounds of rating.
Round 1 Ratings
Overview of Round 1: The primary purpose of Round 1 is to ask the panelists to make their initial
determination as to which achievement level category each portfolio should be classified into. In
this round, panelists will be working individually, without discussion with their colleagues.
Activities:
7. Make sure panelists have the following materials: a. Rating form
b. set of portfolios
c. Achievement Level Descriptors
8. Orient panelists to the set of portfolios. Point out that the portfolios are presented in order,
from lowest scoring to highest. Make sure panelists understand that, even though the
portfolios are presented from lowest- to highest-scoring, their own ratings do not need to be
in strictly increasing order.
9. Starting with the first portfolio, the panelists will review each portfolio in turn. As they are
reviewing the portfolios, the panelists should keep in mind the Achievement Level
Descriptors. They should consider the knowledge, skills, and abilities demonstrated by
each and how they relate to the definitions of the achievement levels. The purpose of this
step is for panelists to make their initial determinations as to how the portfolios should be
Appendix G—Facilitators’ Scripts 11 2010 Maine PAAP Standard Setting Report
categorized into the four achievement levels. The panelists are free to make notes on the
portfolios, sort them into piles, use sticky notes, or use whatever system helps them to keep
track of their categorizations.
10. Panelists may want to take notes as they work if there are particular points they would like
to discuss with their colleagues in Round 2.
a. Have panelists write their ID and round number on the rating form. The ID number
is on their name tags.
b. Briefly remind them how to fill in the rating form.
c. Answer questions the panelists may have about the work in Round 1.
d. Once everyone understands what they are to do in Round 1, tell them to begin.
11. As panelists complete the task, ask them to carefully inspect their rating forms to ensure
they are filled out properly.
a. The ID and round number must be filled in.
b. Each portfolio must be assigned to one and only one achievement level.
c. Reiterate that although the portfolios are presented in order from lowest- to highest-
scoring, the panelists’ category assignments do not need to be in strictly increasing
order.
12. Facilitators should bring all the completed rating forms together to the data analysis work
room for tabulation. Prior to submitting them, however, using a show of hands, indicate on
a piece of chart paper how many panelists assigned each portfolio to each achievement level
category. This chart will be used for the Round 2 discussions.
Tabulation of Round 1 Results Tabulation of Round 1 results will be completed as quickly as possible after receipt of the rating
forms. While the tabulation occurs, the group may take a break.
Appendix G—Facilitators’ Scripts 12 2010 Maine PAAP Standard Setting Report
Round 2 Ratings
Overview of Round 2: In Round 2, the panelists will have an opportunity to discuss their Round 1 placements and to revise their ratings on the basis of that discussion. Prior to beginning the Round
2 discussions, the psychometrician will share the group average cut points based on the Round 1
ratings.
Focusing on any portfolios for which there was disagreement as to how they should be categorized,
the panelists will discuss why they categorized each portfolio as they did, making sure that all
different points of view are included in the discussion.
Once panelists have reviewed and discussed the Round 1 categorizations, the panelists will make
their Round 2 ratings.
Activities:
10. Make sure panelists have the following materials:
d. Rating form
e. set of portfolios
f. Achievement Level Descriptors
11. Have panelists write their ID number and Round 2 on the rating form.
12. Provide an overview of Round 2. Paraphrase the following:
a. As in Round 1, the primary purpose is to categorize each portfolio into the
achievement level category where you believe it belongs.
b. Each panelist needs to base his/her judgments on his/her experience with the
content area, understanding of students, the definition of each achievement level
category, discussions with other panelists, and the knowledge, skills, and abilities
required to answer each item.
13. Show the panelists how the portfolios would be categorized based on the room average
Round 1 cut point placements.
14. Remind panelists that they will be discussing each portfolio with their colleagues, but that
they will be categorizing the portfolios individually. It is not necessary for the panelists to
reach consensus about how to categorize each portfolio.
15. Give panelists an opportunity to ask questions about the task for Round 2.
16. Beginning with the first portfolio for which there is disagreement as to how it should be
categorized, the panelists should begin discussing the categorization of the portfolios
according to the Round 1 ratings.
f. Panelists only need to discuss those portfolios for which there was disagreement as
to how they should be categorized.
g. Panelists should be encouraged to listen to their colleagues as well as express their
own points of view.
h. If the panelists hear a logic/rationale/argument that they did not consider and that
they feel is compelling, then they may adjust their ratings to incorporate that
information.
i. On the basis of the discussions and the feedback presented, panelists should make
their round 2 ratings.
Appendix G—Facilitators’ Scripts 13 2010 Maine PAAP Standard Setting Report
j. The group does not have to achieve consensus. If panelists honestly disagree, that is
fine. We are trying to get the best judgment of each panelist. Panelists should not feel
compelled or coerced into making a rating with which they disagree.
Encourage the panelists to use the discussion and feedback to assess how stringent or
lenient a judge they are. If a panelist is categorizing portfolios consistently higher or
lower than the group, he or she may have a different understanding of the Achievement
Level Descriptors than the rest of the group. It is acceptable for panelists to disagree,
but that disagreement should be based on a common understanding of the
Achievement Level Descriptors.
17. When the group has completed their Round 2 ratings, collect the rating forms. When you
collect the rating forms, carefully inspect them to ensure they are filled out properly.
c. The ID and round number must be filled in.
d. Each portfolio must have one (and only one) rating.
18. Facilitators should bring all the completed rating forms together to the data analysis work
room for tabulation. Prior to submitting them, however, using a show of hands, indicate on
a piece of chart paper how many panelists assigned each portfolio to each achievement level
category. This chart will be used for the Round 3 discussions.
Appendix G—Facilitators’ Scripts 14 2010 Maine PAAP Standard Setting Report
Round 3 Ratings
Overview of Round 3: In Round 3, the panelists will have a final opportunity to discuss their
Round 2 placements and to revise their ratings on the basis of that discussion. Prior to beginning
the Round 3 discussions, the psychometrician will share the Round 2 results with the group,
including the group average cut points and impact data, i.e., the approximate percentage of
students who would be classified into each achievement level category based on the room average
cut points from Round 2.
Once panelists have reviewed and discussed the Round 2 categorizations, they will make their final
ratings.
Activities:
9. Make sure panelists have the following materials:
a. Rating form b. set of portfolios
c. Achievement Level Descriptors
10. Have panelists write their ID number and Round 3 on the rating form.
11. Provide an overview of Round 3. Paraphrase the following:
a. As in Round 2, the primary purpose is to categorize each portfolio into the
achievement level category where you believe it belongs.
b. Each panelist needs to base his/her judgments on his/her experience with the
content area, understanding of students, the definition of each achievement level
category, discussions with other panelists, and the knowledge, skills, and abilities
required to answer each item.
12. Review the feedback information with the panelists. a. Show the panelists how the portfolios would be categorized based on the room
average Round 2 cut point placements.
b. Go over the impact data, explaining that if the Round 2 ratings were to be used to set
the final cut points, these are the approximate percentages of students who would be
classified into each achievement level category.
13. Remind panelists that they will be discussing each portfolio with their colleagues, but that
they will be categorizing the portfolios individually. It is not necessary for the panelists to
reach consensus about how to categorize each portfolio.
14. Give panelists an opportunity to ask questions about the feedback information or about the
task for Round 3.
15. Beginning with the first portfolio for which there is disagreement as to how it should be
categorized, the panelists should begin discussing the categorization of the portfolios
according to the Round 2 ratings.
a. Panelists should discuss the portfolios for which there was disagreement as to how
they should be categorized, focusing in particular on thoe portfolios around the cuts.
b. Panelists should be encouraged to listen to their colleagues as well as express their
own points of view.
c. If the panelists hear a logic/rationale/argument that they did not consider and that
they feel is compelling, then they may adjust their ratings to incorporate that
information.
Appendix G—Facilitators’ Scripts 15 2010 Maine PAAP Standard Setting Report
d. On the basis of the discussions and the feedback presented, panelists should make
their final ratings.
e. The group does not have to achieve consensus. If panelists honestly disagree, that is
fine. We are trying to get the best judgment of each panelist. Panelists should not feel
compelled or coerced into making a rating with which they disagree.
Encourage the panelists to use the discussion and feedback to assess how stringent or
lenient a judge they are. If a panelist is categorizing portfolios consistently higher or
lower than the group, he or she may have a different understanding of the Achievement
Level Descriptors than the rest of the group. It is acceptable for panelists to disagree,
but that disagreement should be based on a common understanding of the
Achievement Level Descriptors.
16. When the group has completed their final ratings, collect the rating forms. When you collect
the rating forms, carefully inspect them to ensure they are filled out properly.
a. The ID and round number must be filled in.
b. Each portfolio for Round 3 must have one (and only one) rating.
c. Return the completed rating forms to the data analysis work room.
Complete Final Evaluation Form After they complete Round 3, have panelists fill out the final evaluation form. Emphasize that their
honest feedback is important.
Appendix G—Facilitators’ Scripts 16 2010 Maine PAAP Standard Setting Report
APPENDIX H—PANELIST AFFILIATIONS
Appendix H—Panelist Affiliations 1 2010 Maine PAAP Standard Setting Report
Armstrong, Linda
Barron, Carla
Belanger, Amanda
Special Educator
Special Education Tchr
RSU 1/Fisher Mitchell
Old Orchard Beach/OOB High School
Union 107/Woodland Elementary School
Reading, 2/3
Math HS
Reading 4/5
Belisle, Mary Math Tchr/Content Specialist SAD 51/Greely Middle School Math 6/7
Boucher, Anne Special Education Tchr RSU 34/Southern Penobscot Regional Program Math 6/7
Butler, Frances
Special Education Tchr
Brewer/Brewer High School
Math HS
Carr, Deb
English Teacher
SAD 55/Sacopee Valley High School
Reading HS
Clark, Jill Special Education Tchr RSU 2/Richmond Middle School Math 6/7
Clemons, Janet Teacher MSAD #15/Gray New Gloucester High School Reading 6/7
Cobb, Patricia
Cole, Emily
Cole, Sally
Special Education Tchr
Elementary Teacher
Reading Interventionist
RSU #11/River View Elementary School
RSU 29/Wellington School
RSU 29/Houlton Elementary School
Reading 4/5
Math 2/3
Reading, 2/3
Coleman, B David English Teacher MSAD 15/Gray-New Gloucester High School Reading HS
Connolly, Kathy
Special Education Tchr
RSU # 26/Glenburn School
Math 6/7
Connolly, Shelley
Special Education Tchr
SAD #4/SAD #4 Elementary School
Math 4/5
Corbett, Terras
Ed Tech 3
Brewer/Brewer High
Math HS
Cornett, Marla Special Education Tchr Lewiston Public Schools/Farwell Math 2/3
Dawson, Daryl Ed Tech 111 Brewer/Brewer High Math HS
Dock, Heidi Special Education Tchr SAD #17/Oxford Elementary Reading, 2/3
Panelist Affiliations
Name Title District/school June 28-29 June 30
Adams, Lynne Asst SPED Director Augusta/Cony Math 2/3 Science 8
Writing 4
Writing 11
Writing 4
Science 8 Science 11 Writing 11
Writing 7
Writing 11
Writing 4
Writing 11 Science 8 Science 5 Science 11
Writing 4
Science 11
Writing 4
continued
Appendix H—Panelist Affiliations 3 2010 Maine PAAP Standard Setting Report
Name Title District/school June 28-29 June 30
Drysdale, Rebekah Special Education Tchr Auburn/Walton Reading 4/5 Writing 7
Dunn, Julie HS SPED Resource Rm MSAD#29/Houlton Reading HS Science 11
Earnhardt, Marge
Frati, Alice
Teacher
Special Education Tchr
Gov. Baxter School f/t Deaf
Bangor School Department/Mary Snow School
Reading 6/7
Math 4/5
Genovese, Katie School Counselor MSAD 15/Gray-New Gloucester Reading HS Science 11
Granger, Sheree Special Education Tchr Sweetser Reading, 2/3
Hargrove, Jesse Special Education Tchr A.O.S. 92/Winslow High School Reading HS Science 11
Hartley, Julie
Hayes, Priscilla
Resource Room Teacher
Special Education Tchr
RSU 11/River View Community School
Auburn/Fairview
Math 4/5
Reading, 2/3
Writing 7
Hayes, Steve Coordinator Special Srvcs Easton School Dept./Easton schools Math 6/7 Writing 11
Herrick, Janet
Hodgkins, Susan
General Education Tchr
Special Services Director
SAD 4/Carroll L. McKusick Elementary
RSU #37/MSAD #37/N/A
Reading, 2/3
Reading 6/7
Writing 7
Howard, Deborah Teacher of the Deaf ME Educational Center f/t Deaf & Hard of Hearing/Governor Baxter School for the Deaf
Reading 4/5 Science 5
Howard, Linda
MSAD 41/Milo Elementary
Math 2/3 Science 5
Inman, Penny Lisbon School Department/Lisbon Community School
Reading, 2/3
Kelley, Debbie
Special Education Tchr
MSAD 37/Narraguagus High School
Math 6/7 Science 8
Lavalle-Rivera, Juan
RSU Science 11
Math 6/7 Science 8
Lessard, Robyn
Special Education Tchr
RSU 24/Ellsworth High School
Reading HS Writing 11
Luginbuhl, Ann Resource Room Teacher Union 104/Charlotte School Math 4/5
Malone, Sean Grade 3-8 Teacher MSAD 14/Union 108/Vanceboro Elementary School
Reading 6/7 Writing 7
McCormick, Kelly Asst Professor of Math University of Southern Maine Math 6/7
continued
Appendix H—Panelist Affiliations 4 2010 Maine PAAP Standard Setting Report
Name Title District/school June 28-29 June 30
Mitchell, Barbara
Special Education Tchr Medway School Department/Medway Middle School
Math 4/5 Science 5
Moody, Lyndon John Calais School Dept - Union 106/Calais High School
Math HS
Mullis, Deborah
Math 2/3 Science 5
Oceipka, Gail
Reeds Brook Middle School
Reading 6/7 Science
8
O'Neill, Kathryn
Peaslee, Kimberly
Pelletier, Deborah
Speech Language Pathologist
Special Education Tchr
Special Educator
RSU 1/ Fisher-Mitchell
MSAD 15/Gray and New Gloucester
Acton Elementary School
Reading 4/5
Reading HS
Reading 6/7
Writing 4
Writing 7
Writing 4
Penner, Nancy Ed Tech Brewer/Brewer Middle School Math HS Science 11
Perry, Heather Plant,
Narda Pomerleau,
Rosemarie Pulkkinen,
Kerri Randall, Lenora
Reed, Paula
Special Educator - Autism
Special Education Tchr
Special Education Tchr
First Grade Teacher
Special Education Tchr
Title 1 Teacher
Lisbon Community School
SAD 20/Fort Fairfield Elem. School
Scarborough/Blue Point
SAD 4/Carroll L. McKusick Elementary
SAD 4/PCHS
Portland/East End Community School
Reading, 2/3
Math 2/3
Reading, 2/3
Reading, 2/3
Math HS
Reading, 2/3
Writing 4
Writing 7
Writing 4
Rehill, Kathy Math 2/3 Science 5
Robbins, Barbara
District Evaluator - SPED
MSAD 74/Solon Elementary
Reading 6/7 Writing
11
Robert, Cheryl
Saponara, Diana
Sawyer, Jane
Seiler, Scott
Sewell, Jill
Special Education Tchr
First Grade Teacher
Special Education Admin.
Special Education Tchr
Special Education Tchr
Lewiston/Farwell
SAD 4/Carroll L. McKusick Elementary
NA/Spring Harbor Academy
MSAD #42/Central Aroostook Jr/Sr High
MSAD #70/Mill Pond School
Math 2/3
Reading 4/5
Math 4/5
Reading 6/7
Reading 4/5
Writing 4
Writing 4
Writing 7
Shardlow, Naomi Retired SPED Director Reading HS Writing
11
continued
Appendix H—Panelist Affiliations 5 2010 Maine PAAP Standard Setting Report
Reading 4/5
Name Title District/school June 28-29 June 30
Skillin, Sarah
Special Education Tchr
Lewiston/Farwell
Math 4/5 Science 5
Stokes, Kelly
Special Educator RSU #15/MSAD #!5/Gray New Gloucester High School
Math HS
Thurber, Jacqueline
Totman, Alice
Tucker, Amy
Special Education Tchr
SPED Tchr - Case Mgr
Technology Integration Specialist
RSU #24/Sumner Memorial High School
RSU 52/Tripp Middle School
RSU 16
Math HS
Math 6/7
Math 6/7
Science 8
Writing 7
Writing 7
Viere, Janet
Special Educator
Auburn School Department/Walton School
Reading 4/5 Writing 11
Vigneault, Rita
RSU #19/Nokomis High School
Math 4/5 Writing 11
Winslow, Susan Title 1 Teacher Portland Public Schools/East End Community
School
Appendix H—Panelist Affiliations 6 2010 Maine PAAP Standard Setting Report
APPENDIX I—EVALUATION RESULTS
Appendix I—Evaluation Results 1 2010 Maine PAAP Standard Setting Report
Mathematics Grades 2 and 3
Mathematics Grade 2 Training Evaluation
N Mean % SD % D % U % A % SA I understand the goals of the standard
setting meeting. 8 4.38 0 0 0 62 38
I understand the procedures we are using
to set standards. 8 3.88 0 0 25 62 12
I understand how to use the standard
setting materials. 8 4.12 0 0 0 88 12
I understand the differences between the
performance levels. 7 3.71 0 0 29 71 0
I understand how to make the cut score
judgment. 7 3.43 0 0 57 43 0
I know what tasks to expect for the
remainder of the meeting. 7 3.29 0 14 43 43 0
I am confident in my understanding of the
standard setting task. 8 3.62 0 0 38 62 0
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please indicate any areas in which you would like more information before you continue.
Cut Scores? Have we discussed that?
Please indicate any questions you may have about the remainder of the standard setting meeting.
I think I understand – not sure
Mathematics Grade 2 Procedural Evaluation
N Mean % SD % D % U % A % SA I understood how to make the cut score
judgments. 8 4.12 0 0 0 88 12
I understood how to use the materials
provided. 8 4.25 0 0 0 75 25
I understood how to record my judgments. 8 4.25 0 0 0 75 25 I think the procedures make sense. 8 3.62 0 25 12 38 25 I am sufficiently familiar with the
assessment. 8 4.25 0 0 12 50 38
I understand the differences between the
performance levels. 8 4 0 12 12 38 38
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please indicate any questions you may have about the remainder of the standard setting meeting.
Very few of the tasks lend themselves to establishing extension to show PWD.
Still struggle with PAAPs incorrectly scored either inaccuracy or in levels of assistance.
N Mean %Low %High The performance level descriptors. 8 4 0 0 12 75 12 The state content standards. 8 4.25 0 0 0 75 25 My perception of the difficulty level of the
assessment. 7 4.14 0 0 14 57 29
The student responses. 8 4.62 0 0 0 38 62 My experience working with students. 8 4.5 0 0 12 25 62
Appendix I—Evaluation Results 3 2010 Maine PAAP Standard Setting Report
What materials, information, or procedures were most influential in your placement of the cut scores?
Why?
Examples included PAAPs with 4 standards- 4th
was counted in standard setting and shouldn’t have
been.
Examples included PAAPs with only 1 task force for a standard- how can you determine
growth/knowledge with only 1 task?
Note: please check Level of Accuracy Data Key, many were incorrect which made the teacher %
incorrect. For instance no 2/3 choice.
The PAAPs themselves. My knowledge of PAAPs. Panel discussions.
Rubric created by our group.
Student work vs. Actual score listed as there were many with incorrectly scored items.
Appendix I—Evaluation Results 4 2010 Maine PAAP Standard Setting Report
Mathematics Grades 2 and 3 Final Evaluation Panelist Demographics N = 8
Gender
Race/ethnicity
Years of experience in education
Area of Expertise (check all that apply)
Male 0 Female 8
White 8 Black 0 Hispanic 0 Asian 0 Pacific Islander 0 American Indian 0
0–5 2 5–10 0 10–15 2 More than 15 4
Students with Disabilities 8
Students with Limited English
Proficiency 0
Economically Disadvantaged
Students 2
Gifted and Talented Students 0
General Education 2
N Mean % SD % D % U % A % SA I understood the goals of the standard
setting meeting. 8 4.38 0 0 0 62 38
I understood the procedures we used to
set standards. 8 4 0 0 25 50 25
The facilitator helped me understand the
process. 8 4.25 0 0 12 50 38
The materials contained the information
needed to set standards. 8 3.38 0 38 12 25 25
I understood how to use the materials
provided. 8 4.12 0 0 12 62 25
The performance level descriptors were
clear. 8 4.25 0 0 12 50 38
I understood how to make the cut score
judgments. 8 3.88 0 12 25 25 38
I understood how to use the feedback
provided after each round. 8 4.12 0 12 0 50 38
I understood how to use the impact data. 7 4 0 0 14 71 14
I understood how the cut scores were
calculated. 8 3.62 12 12 0 50 25
The facilitator was able to get answers to
my questions. 8 4.38 0 0 12 38 50
Sufficient time was allotted for training on
the standard setting tasks. 8 4.25 0 0 12 50 38
Sufficient time was allotted to complete
the standard setting tasks. 8 4.12 0 12 0 50 38
The facilitator helped the standard setting
process run smoothly. 8 4.62 0 0 0 38 62
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Appendix I—Evaluation Results 5 2010 Maine PAAP Standard Setting Report
Influence of N Mean %Low %High The performance level descriptors. 8 4 0 0 12 75 12
Usefulness of N Mean %Low %High The opening session. 8 3.38 12 0 38 38 12 The small group activities. 8 4.38 0 0 0 62 38
Becoming familiar with the assessment. 8 4.12 0 0 12 62 25
Articulating the differences between the
performance levels. 8 3.88 0 0 25 62 12
Discussions with other participants. 8 4.62 0 0 0 38 62
My expectations of students. 8 3.75 0 0 25 75 0 The difficulty of the test materials. 8 4.12 0 0 12 62 25 The student responses. 8 4.25 0 0 0 75 25 My experience in the field. 8 4.12 0 0 25 38 38 Discussions with other participants. 8 4.12 0 12 12 25 50 Cut scores of other participants. 8 3.62 0 12 25 50 12 Impact data. 7 3.14 0 29 29 43 0
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Examples should include those with correctly scored items and correctly identified level of
accuracy marked because we were told they were corrected when Measured Progress scored
but we didn’t have that info.Examples shouldn’t have missing tasks or zeros ( at the upper
limits) makes more difficult to give accurate reading limits) makes more difficult to give
accurate reading. It was unique to see I know that no highest level will create the need for
policy level discussion.
Samples had numerous errors-scoring errors, level of assistance errors, task with scores
below 33 on LOC 4 tasks at “top” of samples (30+) at gr. 3
Better quality of examples. Felt pressured to raise some students’ standings for better cut
scores.
Use tests that are scored correctly. Provide raw scores.
Appendix I—Evaluation Results 6 2010 Maine PAAP Standard Setting Report
Mathematics Grades 4 and 5
Mathematics Grade 4 Training Evaluation
N Mean % SD % D % U % A % SA I understand the goals of the standard
setting meeting. 8 4.75 0 0 0 25 75
I understand the procedures we are using
to set standards. 8 4.75 0 0 0 25 75
I understand how to use the standard
setting materials. 8 4.62 0 0 0 38 62
I understand the differences between the
performance levels. 8 4.12 0 0 0 88 12
I understand how to make the cut score
judgment. 8 4.12 0 0 0 88 12
I know what tasks to expect for the
remainder of the meeting. 8 4.62 0 0 12 12 75
I am confident in my understanding of the
standard setting task. 8 4.5 0 0 0 50 50
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please indicate any areas in which you would like more information before you continue.
Please indicate any questions you may have about the remainder of the standard setting
meeting.
Appendix I—Evaluation Results 7 2010 Maine PAAP Standard Setting Report
N Mean %Low %High The performance level descriptors. 13 4.54 0 0 15 15 69
Mathematics Grade 4 Procedural Evaluation
N Mean % SD % D % U % A % SA I understood how to make the cut score
judgments. 16 4.38 0 0 0 62 38
I understood how to use the materials
provided. 16 4.56 0 0 0 44 56
I understood how to record my judgments. 16 4.75 0 0 0 25 75
I think the procedures make sense. 16 4.44 0 0 6 44 50
I am sufficiently familiar with the
assessment. 16 4.62 0 0 0 38 62
I understand the differences between the
performance levels. 16 4.44 0 0 0 56 44
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please indicate any questions you may have about the remainder of the standard setting
meeting.
The state content standards. 13 3.85 0 0 46 23 31 My perception of the difficulty level of the assessment.
13 3.69 15 0 23 23 38
The student responses. 13 4.46 0 0 15 23 62 My experience working with students. 13 3.92 8 0 23 31 38
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
None were more influential than others
Level of complexity, Level of accuracy, level of assistance
Student responses +my experience
Student work & core skills assessed for grade level
Appendix I—Evaluation Results 8 2010 Maine PAAP Standard Setting Report
Mathematics Grades 4 and 5 Final Evaluation Panelist Demographics N = 8
Gender
Race/ethnicity
Years of experience in education
Area of Expertise (check all that apply)
Male 0 Female 8
White 8 Black 0 Hispanic 0 Asian 0 Pacific Islander 0 American Indian 0
0–5 0 5–10 1 10–15 0 More than 15 7
Students with Disabilities 7
Students with Limited English
Proficiency 1
Economically Disadvantaged
Students 3
Gifted and Talented Students 1
General Education 3
N Mean % SD % D % U % A % SA
I understood the goals of the standard
setting meeting. 8 4.62 0 0 0 38 62
I understood the procedures we used to
set standards. 8 4.62 0 0 0 38 62
The facilitator helped me understand the
process. 8 4.88 0 0 0 12 88
The materials contained the information
needed to set standards. 8 4.62 0 0 0 38 62
I understood how to use the materials
provided. 8 4.62 0 0 0 38 62
The performance level descriptors were
clear. 8 4.62 0 0 0 38 62
I understood how to make the cut score
judgments. 8 4.62 0 0 0 38 62
I understood how to use the feedback
provided after each round. 8 4.75 0 0 0 25 75
I understood how to use the impact data. 8 4.5 0 0 0 50 50
I understood how the cut scores were
calculated. 8 4.5 0 0 0 50 50
The facilitator was able to get answers to
my questions. 8 4.88 0 0 0 12 88
Sufficient time was allotted for training on
the standard setting tasks. 8 4.62 0 0 0 38 62
Sufficient time was allotted to complete
the standard setting tasks. 8 4.25 0 12 12 12 62
The facilitator helped the standard setting
process run smoothly. 8 4.88 0 0 0 12 88
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Appendix I—Evaluation Results 9 2010 Maine PAAP Standard Setting Report
Influence of N Mean %Low %High The performance level descriptors. 8 4.5 0 0 0 50 50
Usefulness of N Mean %Low %High The opening session. 8 3.62 0 25 12 38 25 The small group activities. 8 4.62 0 0 0 38 62
Becoming familiar with the assessment. 8 4.38 0 0 12 38 50
Articulating the differences between the
performance levels. 8 4.62 0 0 0 38 62
Discussions with other participants. 8 4.88 0 0 0 12 88
My expectations of students. 8 4.25 0 0 12 50 38 The difficulty of the test materials. 8 4.38 0 0 12 38 50 The student responses. 8 4.75 0 0 0 25 75 My experience in the field. 8 4.38 0 0 12 38 50 Discussions with other participants. 8 4.62 0 0 0 38 62 Cut scores of other participants. 8 4.25 0 0 12 50 38 Impact data. 8 4.12 0 0 25 38 38
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Sample sizes leading to accuracy was different and impacted decisions.
Super Facilitator
Appendix I—Evaluation Results 10 2010 Maine PAAP Standard Setting Report
Mathematics Grades 6 and 7
Mathematics Grades 6 and 7 Training Evaluation
N Mean % SD % D % U % A % SA I understand the goals of the standard
setting meeting. 10 4.8 0 0 0 20 80
I understand the procedures we are using
to set standards. 10 4.8 0 0 0 20 80
I understand how to use the standard
setting materials. 10 4.7 0 0 10 10 80
I understand the differences between the
performance levels. 10 4.8 0 0 0 20 80
I understand how to make the cut score
judgment. 10 4.1 0 0 30 30 40
I know what tasks to expect for the
remainder of the meeting. 9 4.67 0 0 0 33 67
I am confident in my understanding of the
standard setting task. 10 4.5 0 0 0 50 50
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please indicate any areas in which you would like more information before you continue.
Cut score further explained
Just need to put process to work.
Please indicate any questions you may have about the remainder of the standard setting
meeting.
I believe I will learn what further questions I have from doing the actual work.
None at this time.
Appendix I—Evaluation Results 11 2010 Maine PAAP Standard Setting Report
N Mean %Low %High The performance level descriptors. 10 4.6 0 0 0 40 60
Mathematics Grades 6 and 7 Procedural Evaluation
N Mean % SD % D % U % A % SA I understood how to make the cut score
judgments. 10 4.6 0 0 10 20 70
I understood how to use the materials
provided. 10 4.8 0 0 0 20 80
I understood how to record my judgments. 10 5 0 0 0 0 100
I think the procedures make sense. 10 4.6 0 0 0 40 60
I am sufficiently familiar with the
assessment. 10 4.5 0 0 10 30 60
I understand the differences between the
performance levels. 10 4.5 0 0 0 50 50
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please indicate any questions you may have about the remainder of the standard setting
meeting.
The state content standards. 10 4 0 0 20 60 20 My perception of the difficulty level of assessment.
the 10
4.2
0
0
10
60
30
The student responses. 10 4.7 0 0 0 30 70 My experience working with students. 10 4 0 0 20 60 20
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
The criteria the group set for the steps of the rubric. This help keep me focus when personal
bias came in to play
The combination of LOC & LoA w/ & of accuracy as well as overall & outcome.
Achievement level descriptors helped me to better see subtle difference between levels.
Descriptors
The student PAAP samples and seeing their work (Loc, level of accuracy and level of
assistance). Combining that info with the descriptors helped me place each PAAP.
Defining more specifically the levels of proficiency using the Maine PAAP definitions.
Appendix I—Evaluation Results 12 2010 Maine PAAP Standard Setting Report
Mathematics Grades 6 and 7 Final Evaluation Panelist Demographics N = 10
Gender
Race/ethnicity
Years of experience in education
Area of Expertise (check all that apply)
Male 2 Female 8
White 9 Black 0 Hispanic 1 Asian 0 Pacific Islander 0 American Indian 0
0–5 0 5–10 3 10–15 0 More than 15 7
Students with Disabilities 6
Students with Limited English
Proficiency 0
Economically Disadvantaged
Students 4
Gifted and Talented Students 1
General Education 6
N Mean % SD % D % U % A % SA
I understood the goals of the standard
setting meeting. 10 4.9 0 0 0 10 90
I understood the procedures we used to
set standards. 10 4.8 0 0 0 20 80
The facilitator helped me understand the
process. 10 4.9 0 0 0 10 90
The materials contained the information
needed to set standards. 10 4.8 0 0 0 20 80
I understood how to use the materials
provided. 10 4.8 0 0 0 20 80
The performance level descriptors were
clear. 10 4.6 0 0 0 40 60
I understood how to make the cut score
judgments. 10 4.7 0 0 0 30 70
I understood how to use the feedback
provided after each round. 10 4.7 0 0 0 30 70
I understood how to use the impact data. 10 4.7 0 0 0 30 70
I understood how the cut scores were
calculated. 10 4.6 0 0 0 40 60
The facilitator was able to get answers to
my questions. 10 4.8 0 0 0 20 80
Sufficient time was allotted for training on
the standard setting tasks. 10 4.9 0 0 0 10 90
Sufficient time was allotted to complete
the standard setting tasks. 10 4.9 0 0 0 10 90
The facilitator helped the standard setting
process run smoothly. 10 4.9 0 0 0 10 90
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Appendix I—Evaluation Results 13 2010 Maine PAAP Standard Setting Report
Influence of N Mean %Low %High The performance level descriptors. 10 4.6 0 0 0 40 60
Usefulness of N Mean %Low %High The opening session. 10 3.9 0 10 20 40 30 The small group activities. 10 5 0 0 0 0 100
Becoming familiar with the assessment. 9 4.78 0 0 0 22 78
Articulating the differences between the
performance levels. 10 4.8 0 0 0 20 80
Discussions with other participants. 10 4.9 0 0 0 10 90
My expectations of students. 10 4.2 0 0 30 20 50 The difficulty of the test materials. 10 4.5 0 0 10 30 60 The student responses. 9 5 0 0 0 0 100 My experience in the field. 10 4.6 0 0 0 40 60 Discussions with other participants. 9 4.89 0 0 0 11 89 Cut scores of other participants. 10 4.1 0 10 10 40 40 Impact data. 10 4.1 0 10 10 40 40
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Maybe the intro sessions with lots of repetition + “you’ll learn more about this in small
groups later” could be shorter
Enjoyed the process very much
Appendix I—Evaluation Results 14 2010 Maine PAAP Standard Setting Report
Mathematics Grades 10 and 11
Mathematics Grades 10 and 11 Training Evaluation
N Mean % SD % D % U % A % SA I understand the goals of the standard
setting meeting. 9 4.22 0 0 11 56 33
I understand the procedures we are using
to set standards. 9 4.22 0 0 11 56 33
I understand how to use the standard
setting materials. 9 4.22 0 0 11 56 33
I understand the differences between the
performance levels. 9 4.11 0 0 11 67 22
I understand how to make the cut score
judgment. 9 4 0 0 22 56 22
I know what tasks to expect for the
remainder of the meeting. 9 3.89 0 11 11 56 22
I am confident in my understanding of the
standard setting task. 9 3.67 0 11 22 56 11
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please indicate any areas in which you would like more information before you continue.
Not a question, but a comment, I would have began the achievement levels discussion with
the proficient level instead of the substantially below level proficiency.
How to apply these cuts to the tasks
A bit more time with the materials and I’ll have it I’m almost there.
Let’s go through one together.
Have learned much so far
Please indicate any questions you may have about the remainder of the standard setting
meeting.
I don’t know if I will be able to do this successfully. It seems a little confusing.
I think if we did the first one together, everyone would be on the same page about what we’re
looking for.
Am visual – will feel more confident as one is actually done.
Appendix I—Evaluation Results 15 2010 Maine PAAP Standard Setting Report
N Mean %Low %High The performance level descriptors. 9 4.33 0 0 0 67 33
Mathematics Grades 10 and 11 Procedural Evaluation
N Mean % SD % D % U % A % SA I understood how to make the cut score
judgments. 9 4.11 0 0 11 67 22
I understood how to use the materials
provided. 9 4.22 0 0 0 78 22
I understood how to record my judgments. 9 4.44 0 0 0 56 44
I think the procedures make sense. 9 3.78 0 11 11 67 11
I am sufficiently familiar with the
assessment. 9 4.22 0 0 0 78 22
I understand the differences between the
performance levels. 9 4.22 0 0 0 78 22
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please indicate any questions you may have about the remainder of the standard setting
meeting.
Would have liked to do practice task first.
The state content standards. 9 4.11 0 0 22 44 33 My perception of the difficulty level of t assessment.
he 9
4.33
0
0
0
67
33
The student responses. 9 4 0 0 22 56 22 My experience working with students. 9 3.78 11 0 22 33 33
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
Level descriptions
The descriptions of skills (prior knowledge) in the task descriptions. How specific do the ach.
level descriptions need to be followed when determining the standard?
Difficulty level and content standards
The grade level complexities and how the students performed on those tasks were most
influential to me.
LOC mainly as it loosely corresponded to the skills in the 4 levels.
Appendix I—Evaluation Results 16 2010 Maine PAAP Standard Setting Report
Mathematics Grades 10 and 11 Final Evaluation Panelist Demographics N = 9
Gender
Race/ethnicity
Years of experience in education
Area of Expertise (check all that apply)
Male 2 Female 7
White 9 Black 0 Hispanic 0 Asian 0 Pacific Islander 0 American Indian 0
0–5 0 5–10 2 10–15 3 More than 15 4
Students with Disabilities 8
Students with Limited English
Proficiency 2
Economically Disadvantaged
Students 3
Gifted and Talented Students 0
General Education 2
N Mean % SD % D % U % A % SA
I understood the goals of the standard
setting meeting. 9 4.11 0 11 0 56 33
I understood the procedures we used to
set standards. 9 4 0 11 0 67 22
The facilitator helped me understand the
process. 9 4.22 0 11 0 44 44
The materials contained the information
needed to set standards. 9 3.78 0 0 33 56 11
I understood how to use the materials
provided. 9 4.44 0 0 0 56 44
The performance level descriptors were
clear. 9 3.89 0 0 33 44 22
I understood how to make the cut score
judgments. 9 4.33 0 0 0 67 33
I understood how to use the feedback
provided after each round. 9 4.33 0 0 0 67 33
I understood how to use the impact data. 9 4.11 0 0 11 67 22
I understood how the cut scores were
calculated. 9 3.78 0 11 11 67 11
The facilitator was able to get answers to
my questions. 9 4.33 0 0 0 67 33
Sufficient time was allotted for training on
the standard setting tasks. 9 3.89 0 22 0 44 33
Sufficient time was allotted to complete
the standard setting tasks. 9 3.89 0 11 22 33 33
The facilitator helped the standard setting
process run smoothly. 9 4.44 0 0 0 56 44
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Appendix I—Evaluation Results 17 2010 Maine PAAP Standard Setting Report
Influence of N Mean %Low %High The performance level descriptors. 9 4 0 0 22 56 22
Usefulness of N Mean %Low %High The opening session. 9 2.89 0 22 67 11 0 The small group activities. 9 3.56 11 0 22 56 11
Becoming familiar with the assessment. 9 3.44 0 0 56 44 0
Articulating the differences between the
performance levels. 9 3.78 0 11 11 67 11
Discussions with other participants. 9 3.67 0 11 22 56 11
My expectations of students. 9 3.56 11 0 22 56 11 The difficulty of the test materials. 9 3.89 0 0 33 44 22 The student responses. 9 3.89 0 0 22 67 11 My experience in the field. 9 3.67 22 0 11 22 44 Discussions with other participants. 9 3.22 0 22 33 44 0 Cut scores of other participants. 9 2.44 11 33 56 0 0 Impact data. 9 3.56 0 11 33 44 11
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
I didn’t like the SBP PP P PD skills we listed. I found it was easier to go by LOC’s and
student scores along with the need for assistance.
I think a sample task would be very helpful to go over before starting the evaluations within
your own smaller group.
Process reversed to have open discussion for example of standard setting expectations.
Appendix I—Evaluation Results 18 2010 Maine PAAP Standard Setting Report
Reading Grades 2 and 3
Reading Grades 2 and 3 Training Evaluation
N Mean % SD % D % U % A % SA I understand the goals of the standard
setting meeting. 18 4.17 0 0 11 61 28
I understand the procedures we are using
to set standards. 18 3.94 0 0 22 61 17
I understand how to use the standard
setting materials. 18 3.78 0 0 39 44 17
I understand the differences between the
performance levels. 18 3.83 0 0 33 50 17
I understand how to make the cut score
judgment. 18 3.5 0 0 61 28 11
I know what tasks to expect for the
remainder of the meeting. 18 3.94 0 0 28 50 22
I am confident in my understanding of the
standard setting task. 18 3.56 0 0 56 33 11
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please indicate any areas in which you would like more information before you continue.
Is Proficiency based on how well a student did on this particular assessment or upon where
he should be?
Difference between achievement levels. More about cut scores
Not sure how the indicators we established are helpful when scoring level ½ assessments.
Please indicate any questions you may have about the remainder of the standard setting
meeting.
I just wanted to have the conversation with group to see if we are seeing it in a similar
manner.
Note: It is helpful to have done standard setting before.
How does substantially below proficient and partially proficient affect AYP?
Can a student be considered proficient if they received 100/83 at locs?
Appendix I—Evaluation Results 19 2010 Maine PAAP Standard Setting Report
N Mean %Low %High The performance level descriptors. 11 4 0 0 9 82 9
Reading Grades 2 and 3 Procedural Evaluation
N Mean % SD % D % U % A % SA I understood how to make the cut score
judgments. 11 4.18 0 0 0 82 18
I understood how to use the materials
provided. 11 4.36 0 0 0 64 36
I understood how to record my judgments. 11 4.55 0 0 0 45 55
I think the procedures make sense. 11 4.27 0 0 9 55 36
I am sufficiently familiar with the
assessment. 11 3.91 0 9 18 45 27
I understand the differences between the
performance levels. 11 4.09 0 0 9 73 18
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please indicate any questions you may have about the remainder of the standard setting
meeting.
The state content standards. 11 3.45 0 0 55 45 0 My perception of the difficulty level of assessment.
the 11
4.18
0
0
9
64
27
The student responses. 11 4.18 0 0 9 64 27 My experience working with students. 11 3.82 9 0 0 82 9
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
Understanding of the student’s body of work, level of complexities. I know how difficult it
is for students to complete these tasks and try to give them the benefit of the doubt.
Level of complexity
Level of assistance and achievement level descriptors.
Great facilitators- questions were answered.
Appendix I—Evaluation Results 20 2010 Maine PAAP Standard Setting Report
Reading Grades 2 and 3 Final Evaluation Panelist Demographics N = 11
Gender
Race/ethnicity
Years of experience in education
Area of Expertise (check all that apply)
Male 0 Female 11
White 11 Black 0 Hispanic 0 Asian 0 Pacific Islander 0 American Indian 0
0–5 0 5–10 1 10–15 4 More than 15 6
Students with Disabilities 8
Students with Limited English
Proficiency 1
Economically Disadvantaged
Students 9
Gifted and Talented Students 0
General Education 5
N Mean % SD % D % U % A % SA I understood the goals of the standard
setting meeting. 11 4.36 0 0 0 64 36
I understood the procedures we used to
set standards. 11 4.45 0 0 0 55 45
The facilitator helped me understand the
process. 11 4.36 0 0 9 45 45
The materials contained the information
needed to set standards. 11 4.45 0 0 0 55 45
I understood how to use the materials
provided. 11 4.36 0 0 0 64 36
The performance level descriptors were
clear. 11 4.36 0 0 0 64 36
I understood how to make the cut score
judgments. 11 4.27 0 9 0 45 45
I understood how to use the feedback
provided after each round. 11 4.36 0 0 9 45 45
I understood how to use the impact data. 11 4.09 0 0 27 36 36
I understood how the cut scores were
calculated. 11 4 0 9 18 36 36
The facilitator was able to get answers to
my questions. 11 4.55 0 0 0 45 55
Sufficient time was allotted for training on
the standard setting tasks. 11 4.45 0 0 0 55 45
Sufficient time was allotted to complete
the standard setting tasks. 11 4.45 0 0 0 55 45
The facilitator helped the standard setting
process run smoothly. 11 4.64 0 0 0 36 64
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Appendix I—Evaluation Results 21 2010 Maine PAAP Standard Setting Report
Influence of N Mean %Low %High The performance level descriptors. 11 4.36 0 0 0 64 36
Usefulness of N Mean %Low %High The opening session. 10 3.8 10 0 20 40 30 The small group activities. 11 4.36 0 0 9 45 45
Becoming familiar with the assessment. 11 4.45 0 0 9 36 55
Articulating the differences between the
performance levels. 10 4.5 0 0 0 50 50
Discussions with other participants. 11 4.64 0 0 0 36 64
My expectations of students. 11 4.27 0 0 9 55 36 The difficulty of the test materials. 11 4.36 0 0 9 45 45 The student responses. 11 4.27 0 0 0 73 27 My experience in the field. 11 4 0 0 27 45 27 Discussions with other participants. 11 4.45 0 0 9 36 55 Cut scores of other participants. 11 3.82 0 0 45 27 27 Impact data. 11 4 0 0 18 64 18
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Appendix I—Evaluation Results 22 2010 Maine PAAP Standard Setting Report
Reading Grades 4 and 5
Reading Grades 4 and 5 Training Evaluation
N Mean % SD % D % U % A % SA I understand the goals of the standard
setting meeting. 9 4.22 0 0 11 56 33
I understand the procedures we are using
to set standards. 9 4.11 0 0 11 67 22
I understand how to use the standard
setting materials. 9 4 0 0 11 78 11
I understand the differences between the
performance levels. 8 4.12 0 0 0 88 12
I understand how to make the cut score
judgment. 9 3.33 0 0 78 11 11
I know what tasks to expect for the
remainder of the meeting. 9 4.11 0 0 11 67 22
I am confident in my understanding of the
standard setting task. 9 3.78 0 0 33 56 11
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please indicate any areas in which you would like more information before you continue.
As a reg. ed Teacher I wish I had prepared myself more on PAAPs because I have realized
that I really knew very little. Thank you for the brief introduction.
The training provided was sequential and with a clearly stated outcome. At this point, I need
to get my feet wet with the process. Maybe ?’s will arise as I am completing the task.
Example of how to score demonstrated would be helpful on first task as a group.
Please indicate any questions you may have about the remainder of the standard setting
meeting.
Appendix I—Evaluation Results 23 2010 Maine PAAP Standard Setting Report
N Mean %Low %High The performance level descriptors. 9 4 0 0 11 78 11
Reading Grades 4 and 5 Procedural Evaluation
N Mean % SD % D % U % A % SA I understood how to make the cut score
judgments. 9 4.11 0 0 11 67 22
I understood how to use the materials
provided. 9 4.33 0 0 0 67 33
I understood how to record my judgments. 9 4.44 0 0 0 56 44
I think the procedures make sense. 9 4.22 0 0 11 56 33
I am sufficiently familiar with the
assessment. 9 3.89 0 0 33 44 22
I understand the differences between the
performance levels. 9 4.11 0 0 11 67 22
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please indicate any questions you may have about the remainder of the standard setting
meeting.
The procedures make sense but I feel like we could have had more time. Explanations of
different ability levels would have been helpful for Regular Ed. Teachers.
The state content standards. 9 3.56 11 0 22 56 11 My perception of the difficulty level of t assessment.
he 9
3.44
0
11
33
56
0
The student responses. 9 4.22 0 0 0 78 22 My experience working with students. 9 4.33 0 0 11 44 44
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
Discussion- as a reg. ed teacher I expected the students to be able to read for the reading
section but they didn’t have to. Hard not to compare to the NECAP.
Discussion among facilitator and group; more time should be allowed for this process- felt
very rushed.
The more info we have about student achievement, the better decisions we can make.
Discussions were helpful; did feel rushed at times to get through the stack!
LOC; level of assistance; for successive rounds, listening to others’ feedback
LOCs and the student work with score sheets; more info (specific info) would have helped
with regards to the level of assistance and actual students responses, not just correct or
incorrect.
Appendix I—Evaluation Results 24 2010 Maine PAAP Standard Setting Report
Having the handouts which defined the Achievement Levels as well as the charts with
additional descriptors were useful because I didn’t have to rely on my memory for the info- I
had a reference(s) to go back to.
I would have appreciated more level of assistance descriptions from teachers. As a teacher
who has given PAAPs I was never made aware of the formula in scoring the different levels
of the LOC and the weight given to the higher levels in scoring.
The LOC Indicators were very helpful with this process. I also would have like to have more
info for LO Assistance from teachers.
Appendix I—Evaluation Results 25 2010 Maine PAAP Standard Setting Report
Reading Grades 4 and 5 Final Evaluation Panelist Demographics N = 9
Gender
Race/ethnicity
Years of experience in education
Area of Expertise (check all that apply)
Male 0 Female 9
White 9 Black 0 Hispanic 0 Asian 0 Pacific Islander 0 American Indian 0
0–5 1 5–10 2 10–15 1 More than 15 5
Students with Disabilities 6
Students with Limited English
Proficiency 2
Economically Disadvantaged
Students 3
Gifted and Talented Students 0
General Education 5
N Mean % SD % D % U % A % SA
I understood the goals of the standard
setting meeting. 9 4.22 0 0 0 78 22
I understood the procedures we used to
set standards. 9 4.22 0 0 11 56 33
The facilitator helped me understand the
process. 9 4.11 0 0 11 67 22
The materials contained the information
needed to set standards. 9 4 0 0 11 78 11
I understood how to use the materials
provided. 8 4.25 0 0 0 75 25
The performance level descriptors were
clear. 9 4 0 0 11 78 11
I understood how to make the cut score
judgments. 9 3.67 0 0 33 67 0
I understood how to use the feedback
provided after each round. 9 4.33 0 0 0 67 33
I understood how to use the impact data. 9 3.56 0 0 56 33 11
I understood how the cut scores were
calculated. 8 3.12 0 25 38 38 0
The facilitator was able to get answers to
my questions. 9 4.44 0 0 0 56 44
Sufficient time was allotted for training on
the standard setting tasks. 9 3.89 0 11 11 56 22
Sufficient time was allotted to complete
the standard setting tasks. 9 3.89 0 11 11 56 22
The facilitator helped the standard setting
process run smoothly. 9 4.44 0 0 0 56 44
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Appendix I—Evaluation Results 26 2010 Maine PAAP Standard Setting Report
Influence of N Mean %Low %High The performance level descriptors. 9 4 0 0 11 78 11
Usefulness of N Mean %Low %High The opening session. 9 3.22 0 22 44 22 11 The small group activities. 9 4 0 0 22 56 22
Becoming familiar with the assessment. 9 3.67 0 11 22 56 11
Articulating the differences between the
performance levels. 9 4 0 0 22 56 22
Discussions with other participants. 9 4.78 0 0 0 22 78
My expectations of students. 9 4.11 0 0 11 67 22 The difficulty of the test materials. 9 4 0 0 11 78 11 The student responses. 9 4.22 0 0 0 78 22 My experience in the field. 9 4.33 0 0 0 67 33 Discussions with other participants. 9 4.56 0 0 0 44 56 Cut scores of other participants. 9 3.56 0 0 44 56 0 Impact data. 9 3 11 0 67 22 0
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Devise some sort of graphic organizer to record notes as we revised the portfolios (to keep
track of our reactions + inputs)
Appendix I—Evaluation Results 27 2010 Maine PAAP Standard Setting Report
Reading Grades 6 and 7
Reading Grades 6 and 7 Training Evaluation
N Mean % SD % D % U % A % SA I understand the goals of the standard
setting meeting. 8 4.38 0 0 0 62 38
I understand the procedures we are using
to set standards. 8 4.12 0 12 0 50 38
I understand how to use the standard
setting materials. 8 4.12 0 0 12 62 25
I understand the differences between the
performance levels. 8 4.38 0 0 0 62 38
I understand how to make the cut score
judgment. 8 4.12 0 0 0 88 12
I know what tasks to expect for the
remainder of the meeting. 8 4.12 0 0 0 88 12
I am confident in my understanding of the
standard setting task. 8 3.88 0 12 0 75 12
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please indicate any areas in which you would like more information before you continue.
Cut score
Please indicate any questions you may have about the remainder of the standard setting
meeting.
Appendix I—Evaluation Results 28 2010 Maine PAAP Standard Setting Report
N Mean %Low %High The performance level descriptors. 8 2.5 12 25 62 0 0
Reading Grades 6 and 7 Procedural Evaluation
N Mean % SD % D % U % A % SA I understood how to make the cut score
judgments. 8 3.62 0 0 38 62 0
I understood how to use the materials
provided. 8 3.5 0 12 25 62 0
I understood how to record my judgments. 8 4 0 0 12 75 12
I think the procedures make sense. 8 2.62 12 25 50 12 0
I am sufficiently familiar with the
assessment. 7 4.29 0 0 0 71 29
I understand the differences between the
performance levels. 8 3.62 0 12 25 50 12
Please indicate any questions you may have about the remainder of the standard setting
meeting.
The state content standards. 8 3 0 25 62 0 12 My perception of the difficulty level of the assessment.
8 3.12 0 0 88 12 0
The student responses. 8 4.25 0 0 12 50 38 My experience working with students. 8 3.75 0 0 38 50 12
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
Using the LOCs and TLC evidence of growth influences my cut score rating.
The LOC levels
It appears as if the Loc’s do not match the tasks. Ex an A1 LOC the student must be using
phoenic awareness/word parts and context clues, yet the task does not require the student
read. I feel this needs to be examined more closely.
It became obvious during/prior to Round 3 what Measured Progress was “looking for” this
influenced the decisions made causing me to question the validity of this entire process.
Difficulty of the task along with student responses.
Appendix I—Evaluation Results 29 2010 Maine PAAP Standard Setting Report
Reading Grades 6 and 7 Final Evaluation Panelist Demographics N = 8
Gender
Race/ethnicity
Years of experience in education
Area of Expertise (check all that apply)
Male 2 Female 6
White 8 Black 0 Hispanic 0 Asian 0 Pacific Islander 0 American Indian 0
0–5 1 5–10 1 10–15 0 More than 15 6
Students with Disabilities 6
Students with Limited English
Proficiency 2
Economically Disadvantaged
Students 5
Gifted and Talented Students 1
General Education 4
N Mean % SD % D % U % A % SA
I understood the goals of the standard
setting meeting. 8 4 0 0 0 100 0
I understood the procedures we used to
set standards. 8 3.75 0 0 25 75 0
The facilitator helped me understand the
process. 8 3.88 0 0 12 88 0
The materials contained the information
needed to set standards. 8 3.62 0 0 50 38 12
I understood how to use the materials
provided. 8 3.88 0 0 12 88 0
The performance level descriptors were
clear. 8 3.38 0 0 62 38 0
I understood how to make the cut score
judgments. 7 3.71 0 0 29 71 0
I understood how to use the feedback
provided after each round. 8 3.88 0 0 12 88 0
I understood how to use the impact data. 8 3.75 0 0 25 75 0
I understood how the cut scores were
calculated. 8 3.75 0 0 25 75 0
The facilitator was able to get answers to
my questions. 8 3.75 0 0 25 75 0
Sufficient time was allotted for training on
the standard setting tasks. 8 4.25 0 0 0 75 25
Sufficient time was allotted to complete
the standard setting tasks. 8 4.25 0 0 0 75 25
The facilitator helped the standard setting
process run smoothly. 8 3.88 0 0 25 62 12
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Appendix I—Evaluation Results 30 2010 Maine PAAP Standard Setting Report
Usefulness of N Mean %Low %High The opening session. 8 3.38 0 12 50 25 12
Influence of N Mean %Low %High The performance level descriptors. 8 3.12 0 12 62 25 0
The small group activities. 8 3.88 0 0 25 62 12 Becoming familiar with the assessment. 8 3.88 0 0 25 62 12 Articulating the differences between the performance levels.
8 3.75 0 0 25 75 0
Discussions with other participants. 8 4.38 0 0 12 38 50 Providing additional details to the performance level descriptors.
2 4 0 0 0 100 0
My expectations of students. 8 3.75 0 0 25 75 0 The difficulty of the test materials. 8 3.62 0 0 38 62 0 The student responses. 8 3.88 0 0 25 62 12 My experience in the field. 8 3.88 0 0 25 62 12 Discussions with other participants. 8 4 0 0 12 75 12 Cut scores of other participants. 8 3.38 0 0 62 38 0 Impact data. 8 3.62 0 12 12 75 0
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
The achievement level definitions made this process difficult. They do not match the tasks,
therefore made this process mute, in my opinion.
Appendix I—Evaluation Results 31 2010 Maine PAAP Standard Setting Report
Reading Grades 10 and 11
Reading Grades 10 and 11 Training Evaluation
N Mean % SD % D % U % A % SA I understand the goals of the standard
setting meeting. 7 4.57 0 0 0 43 57
I understand the procedures we are using
to set standards. 7 3.86 0 0 14 86 0
I understand how to use the standard
setting materials. 6 4 0 0 0 100 0
I understand the differences between the
performance levels. 7 4 0 0 14 71 14
I understand how to make the cut score
judgment. 7 3.71 0 0 29 71 0
I know what tasks to expect for the
remainder of the meeting. 7 4.29 0 0 0 71 29
I am confident in my understanding of the
standard setting task. 7 3.86 0 0 29 57 14
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please indicate any areas in which you would like more information before you continue.
Please indicate any questions you may have about the remainder of the standard setting
meeting.
Appendix I—Evaluation Results 32 2010 Maine PAAP Standard Setting Report
N Mean %Low %High The performance level descriptors. 7 4.71 0 0 0 29 71
Reading Grades 10 and 11 Procedural Evaluation
N Mean % SD % D % U % A % SA I understood how to make the cut score
judgments. 7 4.43 0 0 0 57 43
I understood how to use the materials
provided. 7 4.71 0 0 0 29 71
I understood how to record my judgments. 7 4.71 0 0 0 29 71
I think the procedures make sense. 7 4.43 0 0 0 57 43
I am sufficiently familiar with the
assessment. 7 4.57 0 0 0 43 57
I understand the differences between the
performance levels. 7 4.43 0 0 0 57 43
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please indicate any questions you may have about the remainder of the standard setting
meeting.
(referring to “I think the procedures make sense”; subject checked “agree”): This seems
counter!
Subjectivity?
The state content standards. 7 4 0 14 0 57 29 My perception of the difficulty level of the assessment.
7 4.71 0 0 0 29 71
The student responses. 7 4.43 0 0 0 57 43 My experience working with students. 7 4.29 0 0 14 43 43
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
Group discussion about student work. Initial setting of achievement indicators.
Fantastic process.
Group discussions and skills at each level.
Achievement Level Descriptors, definitions of the four levels- handout
Appendix I—Evaluation Results 33 2010 Maine PAAP Standard Setting Report
N Mean %Low %High The performance level descriptors. 7 4.71 0 0 0 29 71
Reading Grades 10 and 11 Final Evaluation
N Mean % SD % D % U % A % SA I understood how to make the cut score
judgments. 7 4.57 0 0 0 43 57
I understood how to use the materials
provided. 7 4.71 0 0 0 29 71
I understood how to record my judgments. 7 4.71 0 0 0 29 71
I think the procedures make sense. 7 4.43 0 0 0 57 43
I am sufficiently familiar with the
assessment. 7 4.43 0 0 0 57 43
I understand the differences between the
performance levels. 7 4.29 0 0 0 71 29
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please indicate any questions you may have about the remainder of the standard setting
meeting.
The state content standards. 7 4.14 0 14 0 43 43 My perception of the difficulty level of t assessment.
he 7
4.57
0
0
0
43
57
The student responses. 7 4.57 0 0 0 43 57 My experience working with students. 7 4.43 0 0 14 29 57
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
Excellent Process
Description of 4 levels SBP,PP,P,PWD + achievement level descriptors
The discussions and standards
Student work and discussion about it. Achievement indicators
Appendix I—Evaluation Results 34 2010 Maine PAAP Standard Setting Report
Science Grade 5
Gender
Science Grade 5 Final Evaluation Panelist Demographics N = 7
Male 0 Female 7
White 7
Black 0
Race/ethnicity
Years of experience in education
Area of Expertise (check all that apply)
Hispanic 0 Asian 0 Pacific Islander 0 American Indian 0
0–5 0 5–10 1 10–15 2 More than 15 4
Students with Disabilities 7
Students with Limited English
Proficiency 1
Economically Disadvantaged
Students 1
Gifted and Talented Students 0
General Education 3
N Mean % SD % D % U % A % SA
I understood the goals of the standard
setting meeting. 7 4.57 0 0 0 43 57
I understood the procedures we used to
set standards. 7 4.43 0 0 14 29 57
The facilitator helped me understand the
process. 7 4.57 0 0 0 43 57
The materials contained the information
needed to set standards. 7 4.57 0 0 0 43 57
I understood how to use the materials
provided. 7 4.57 0 0 0 43 57
The performance level descriptors were
clear. 7 4.57 0 0 0 43 57
I understood how to make the cut score
judgments. 7 4.43 0 0 14 29 57
I understood how to use the feedback
provided after each round. 7 4.57 0 0 0 43 57
I understood how to use the impact data. 7 4.57 0 0 0 43 57
I understood how the cut scores were
calculated. 7 4.14 0 0 29 29 43
The facilitator was able to get answers to
my questions. 7 4.71 0 0 0 29 71
Sufficient time was allotted for training on
the standard setting tasks. 7 4.57 0 0 0 43 57
Sufficient time was allotted to complete
the standard setting tasks. 7 4.57 0 0 0 43 57
The facilitator helped the standard setting
process run smoothly. 7 4.57 0 0 0 43 57
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Appendix I—Evaluation Results 35 2010 Maine PAAP Standard Setting Report
Influence of N Mean %Low %High The performance level descriptors. 7 4.71 0 0 0 29 71
Usefulness of N Mean %Low %High The opening session. 7 4.14 0 0 14 57 29 The small group activities. 7 4.71 0 0 0 29 71
Becoming familiar with the assessment. 7 4.29 0 0 14 43 43
Articulating the differences between the
performance levels. 7 4.71 0 0 0 29 71
Discussions with other participants. 7 4.71 0 0 0 29 71
My expectations of students. 7 4.29 0 0 14 43 43 The difficulty of the test materials. 7 4.29 0 0 14 43 43 The student responses. 7 4.86 0 0 0 14 86 My experience in the field. 7 4.86 0 0 0 14 86 Discussions with other participants. 7 4.71 0 0 0 29 71 Cut scores of other participants. 7 4.14 0 14 0 43 43 Impact data. 7 4.29 0 0 14 43 43
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Many teacher still are not following the process-fair amount of teacher error rather than
student error
It was more difficult than need to due to lack of higher LOC’s at some indicators
Much better samples than for math!
( Referring to “I understood how to make cut score judgments.”) Seriously, I don’t feel I
make cut score judgments
If this were to be done again, it would be helpful to have a summary sheet for each students
with task , LOL,LOA for all tasks on 1 sheet. This would save a lot of time, also ever
consider going paperless. Some of this could definitely be paperless
Appendix I—Evaluation Results 36 2010 Maine PAAP Standard Setting Report
Science Grade 8
Gender
Science Grade 8 Final Evaluation Panelist Demographics N = 7
Male 1 Female 6
White 6
Black 0
Race/ethnicity
Years of experience in education
Area of Expertise (check all that apply)
Hispanic 1 Asian 0 Pacific Islander 0 American Indian 0
0–5 0 5–10 0 10–15 0 More than 15 7
Students with Disabilities 7
Students with Limited English
Proficiency 0
Economically Disadvantaged
Students 2
Gifted and Talented Students 0
General Education 2
N Mean % SD % D % U % A % SA
I understood the goals of the standard
setting meeting. 7 4.71 0 0 0 29 71
I understood the procedures we used to
set standards. 7 4.71 0 0 0 29 71
The facilitator helped me understand the
process. 7 4.43 0 0 14 29 57
The materials contained the information
needed to set standards. 7 4.86 0 0 0 14 86
I understood how to use the materials
provided. 7 4.86 0 0 0 14 86
The performance level descriptors were
clear. 7 4.57 0 0 14 14 71
I understood how to make the cut score
judgments. 7 4.57 0 0 0 43 57
I understood how to use the feedback
provided after each round. 7 4.57 0 0 0 43 57
I understood how to use the impact data. 7 4.57 0 0 0 43 57
I understood how the cut scores were
calculated. 7 4.43 0 0 14 29 57
The facilitator was able to get answers to
my questions. 7 4.86 0 0 0 14 86
Sufficient time was allotted for training on
the standard setting tasks. 7 4.86 0 0 0 14 86
Sufficient time was allotted to complete
the standard setting tasks. 7 4.86 0 0 0 14 86
The facilitator helped the standard setting
process run smoothly. 7 4.71 0 0 0 29 71
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Appendix I—Evaluation Results 37 2010 Maine PAAP Standard Setting Report
Influence of N Mean %Low %High The performance level descriptors. 7 4.71 0 0 0 29 71
Usefulness of N Mean %Low %High The opening session. 7 4 14 0 0 43 43 The small group activities. 7 4.57 0 0 0 43 57
Becoming familiar with the assessment. 7 4.43 0 0 14 29 57
Articulating the differences between the
performance levels. 7 4.86 0 0 0 14 86
Discussions with other participants. 7 5 0 0 0 0 100
Providing additional details to the
performance level descriptors. 4 5 0 0 0 0 100
My expectations of students. 7 4.14 0 0 14 57 29 The difficulty of the test materials. 7 3.86 0 0 29 57 14 The student responses. 7 4.71 0 0 14 0 86 My experience in the field. 7 4 0 14 0 57 29 Discussions with other participants. 7 5 0 0 0 0 100 Cut scores of other participants. 7 3.86 14 0 0 57 29 Impact data. 7 4 0 0 14 71 14
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
I enjoyed the process very much.
Hope the blue print and fall trainings can give clarity to next years grade level expectations
and that a full task bank will provide more evaluation options.
Appendix I—Evaluation Results 38 2010 Maine PAAP Standard Setting Report
Science Grade 11
Gender
Science Grade 11 Final Evaluation Panelist Demographics N = 7
Male 2 Female 5
White 7
Black 0
Race/ethnicity
Years of experience in education
Area of Expertise (check all that apply)
Hispanic 0 Asian 0 Pacific Islander 0 American Indian 0
0–5 1 5–10 1 10–15 4 More than 15 1
Students with Disabilities 7
Students with Limited English
Proficiency 1
Economically Disadvantaged
Students 1
Gifted and Talented Students 1
General Education 2
N Mean % SD % D % U % A % SA
I understood the goals of the standard
setting meeting. 7 4.29 0 0 0 71 29
I understood the procedures we used to
set standards. 7 4 0 0 14 71 14
The facilitator helped me understand the
process. 7 4.14 0 0 0 86 14
The materials contained the information
needed to set standards. 7 4 0 0 29 43 29
I understood how to use the materials
provided. 7 4.14 0 0 14 57 29
The performance level descriptors were
clear. 7 3.71 0 0 29 71 0
I understood how to make the cut score
judgments. 7 4 0 0 14 71 14
I understood how to use the feedback
provided after each round. 7 4.14 0 0 14 57 29
I understood how to use the impact data. 7 4.29 0 0 0 71 29
I understood how the cut scores were
calculated. 7 4.29 0 0 0 71 29
The facilitator was able to get answers to
my questions. 7 4.14 0 0 0 86 14
Sufficient time was allotted for training on
the standard setting tasks. 7 4.29 0 0 14 43 43
Sufficient time was allotted to complete
the standard setting tasks. 7 4.43 0 0 0 57 43
The facilitator helped the standard setting
process run smoothly. 7 4.29 0 0 0 71 29
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Appendix I—Evaluation Results 39 2010 Maine PAAP Standard Setting Report
Influence of N Mean %Low %High The performance level descriptors. 7 3.86 0 0 29 57 14
Usefulness of N Mean %Low %High The opening session. 5 3.4 0 20 40 20 20 The small group activities. 7 3.71 0 0 43 43 14
Becoming familiar with the assessment. 7 4 0 0 29 43 29
Articulating the differences between the
performance levels. 7 3.86 0 0 14 86 0
Discussions with other participants. 7 4.14 0 0 29 29 43
Providing additional details to the
performance level descriptors. 1 4 0 0 0 100 0
My expectations of students. 7 4 0 0 29 43 29 The difficulty of the test materials. 7 4 0 0 14 71 14 The student responses. 7 4.14 0 0 14 57 29 My experience in the field. 7 4.14 0 0 29 29 43 Discussions with other participants. 7 4.14 0 0 29 29 43 Cut scores of other participants. 7 3.71 0 0 43 43 14 Impact data. 7 3.86 0 0 29 57 14
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Process between facilitators seemed at times different. Possible that this is due to different
groups? Content areas.
Appendix I—Evaluation Results 40 2010 Maine PAAP Standard Setting Report
Writing Grade 4
Gender
Writing Grade 4 Final Evaluation Panelist Demographics N = 10
Male 0 Female 10
White 10
Black 0
Race/ethnicity
Years of experience in education
Area of Expertise (check all that apply)
Hispanic 0 Asian 0 Pacific Islander 0 American Indian 0
0–5 1 5–10 3 10–15 2 More than 15 4
Students with Disabilities 8
Students with Limited English
Proficiency 0
Economically Disadvantaged
Students 6
Gifted and Talented Students 0
General Education 3
N Mean % SD % D % U % A % SA
I understood the goals of the standard
setting meeting. 9 4.44 0 0 0 56 44
I understood the procedures we used to
set standards. 9 4.67 0 0 0 33 67
The facilitator helped me understand the
process. 9 4.56 0 0 0 44 56
The materials contained the information
needed to set standards. 9 4.56 0 0 0 44 56
I understood how to use the materials
provided. 9 4.67 0 0 0 33 67
The performance level descriptors were
clear. 9 4.67 0 0 0 33 67
I understood how to make the cut score
judgments. 9 4.67 0 0 0 33 67
I understood how to use the feedback
provided after each round. 9 4.56 0 0 0 44 56
I understood how to use the impact data. 9 4.44 0 0 0 56 44
I understood how the cut scores were
calculated. 10 4.2 0 0 20 40 40
The facilitator was able to get answers to
my questions. 9 4.67 0 0 0 33 67
Sufficient time was allotted for training on
the standard setting tasks. 9 4.56 0 0 0 44 56
Sufficient time was allotted to complete
the standard setting tasks. 9 4.67 0 0 0 33 67
The facilitator helped the standard setting
process run smoothly. 9 4.78 0 0 0 22 78
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Appendix I—Evaluation Results 41 2010 Maine PAAP Standard Setting Report
Influence of N Mean %Low %High The performance level descriptors. 10 4.4 0 0 0 60 40
Usefulness of N Mean %Low %High The opening session. 3 3.67 0 0 33 67 0 The small group activities. 8 4.88 0 0 0 12 88
Becoming familiar with the assessment. 9 4.78 0 0 0 22 78
Articulating the differences between the
performance levels. 8 4.88 0 0 0 12 88
Discussions with other participants. 9 5 0 0 0 0 100
Providing additional details to the
performance level descriptors. 4 5 0 0 0 0 100
My expectations of students. 10 4.2 0 0 0 80 20 The difficulty of the test materials. 10 4.3 0 0 0 70 30 The student responses. 10 4.7 0 0 0 30 70 My experience in the field. 10 4.3 0 0 0 70 30 Discussions with other participants. 9 4.67 0 0 0 33 67 Cut scores of other participants. 10 3.9 0 0 30 50 20 Impact data. 10 3.9 0 0 30 50 20
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Since I was unfamiliar with what the student was expected to do, it would have been more
helpful to see the tasks before we designed our rubrics.
Appendix I—Evaluation Results 42 2010 Maine PAAP Standard Setting Report
Writing Grade 7
Gender
Writing Grade 7 Final Evaluation Panelist Demographics N = 9
Male 1 Female 8
White 9
Black 0
Race/ethnicity
Years of experience in education
Area of Expertise (check all that apply)
Hispanic 0 Asian 0 Pacific Islander 0 American Indian 0
0–5 1 5–10 2 10–15 0 More than 15 5
Students with Disabilities 5
Students with Limited English
Proficiency 0
Economically Disadvantaged
Students 3
Gifted and Talented Students 0
General Education 5
N Mean % SD % D % U % A % SA
I understood the goals of the standard
setting meeting. 9 4.67 0 0 0 33 67
I understood the procedures we used to
set standards. 9 4.67 0 0 0 33 67
The facilitator helped me understand the
process. 9 4.67 0 0 0 33 67
The materials contained the information
needed to set standards. 9 4.67 0 0 0 33 67
I understood how to use the materials
provided. 9 4.67 0 0 0 33 67
The performance level descriptors were
clear. 9 4.56 0 0 0 44 56
I understood how to make the cut score
judgments. 9 4.56 0 0 0 44 56
I understood how to use the feedback
provided after each round. 9 4.56 0 0 0 44 56
I understood how to use the impact data. 9 4.56 0 0 0 44 56
I understood how the cut scores were
calculated. 9 4.56 0 0 0 44 56
The facilitator was able to get answers to
my questions. 9 4.67 0 0 0 33 67
Sufficient time was allotted for training on
the standard setting tasks. 9 4.67 0 0 0 33 67
Sufficient time was allotted to complete
the standard setting tasks. 9 4.67 0 0 0 33 67
The facilitator helped the standard setting
process run smoothly. 9 4.67 0 0 0 33 67
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Appendix I—Evaluation Results 43 2010 Maine PAAP Standard Setting Report
Influence of N Mean %Low %High The performance level descriptors. 9 4 0 0 22 56 22
Usefulness of N Mean %Low %High The opening session. 8 3.75 0 0 38 50 12 The small group activities. 9 4.33 0 0 11 44 44
Becoming familiar with the assessment. 9 4.22 0 0 11 56 33
Articulating the differences between the
performance levels. 9 4.56 0 0 0 44 56
Discussions with other participants. 9 4.67 0 0 0 33 67
Providing additional details to the
performance level descriptors. 1 4 0 0 0 100 0
My expectations of students. 9 4.11 0 0 22 44 33 The difficulty of the test materials. 9 4.11 0 0 22 44 33 The student responses. 8 4.62 0 0 0 38 62 My experience in the field. 9 4.11 0 0 22 44 33 Discussions with other participants. 9 4.67 0 0 0 33 67 Cut scores of other participants. 9 4 0 11 11 44 33 Impact data. 8 4.25 0 0 12 50 38
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Awesome 3 days! This is the best I have been to in 16 yrs!
Appendix I—Evaluation Results 44 2010 Maine PAAP Standard Setting Report
Writing Grade 11
Gender
Writing Grade 11 Final Evaluation Panelist Demographics N = 8
Male 2 Female 6
White 8
Black 0
Race/ethnicity
Years of experience in education
Area of Expertise (check all that apply)
Hispanic 0 Asian 0 Pacific Islander 0 American Indian 0
0–5 0 5–10 0 10–15 1 More than 15 7
Students with Disabilities 5
Students with Limited English
Proficiency 1
Economically Disadvantaged
Students 6
Gifted and Talented Students 2
General Education 4
N Mean % SD % D % U % A % SA
I understood the goals of the standard
setting meeting. 8 4.25 0 0 0 75 25
I understood the procedures we used to
set standards. 8 4.25 0 0 0 75 25
The facilitator helped me understand the
process. 8 4.38 0 0 0 62 38
The materials contained the information
needed to set standards. 8 4 0 0 25 50 25
I understood how to use the materials
provided. 8 4.25 0 0 0 75 25
The performance level descriptors were
clear. 8 4.12 0 0 12 62 25
I understood how to make the cut score
judgments. 8 4.25 0 0 0 75 25
I understood how to use the feedback
provided after each round. 8 4.25 0 0 0 75 25
I understood how to use the impact data. 8 4 0 0 25 50 25
I understood how the cut scores were
calculated. 8 4 0 12 0 62 25
The facilitator was able to get answers to
my questions. 8 4.38 0 0 0 62 38
Sufficient time was allotted for training on
the standard setting tasks. 8 4.25 0 0 0 75 25
Sufficient time was allotted to complete
the standard setting tasks. 8 4 0 12 0 62 25
The facilitator helped the standard setting
process run smoothly. 8 4.5 0 0 0 50 50
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Appendix I—Evaluation Results 45 2010 Maine PAAP Standard Setting Report
Influence of N Mean %Low %High The performance level descriptors. 8 4.38 0 0 0 62 38
Usefulness of N Mean %Low %High The opening session. 6 3.67 0 17 0 83 0 The small group activities. 8 4 0 0 12 75 12
Becoming familiar with the assessment. 8 4 0 0 12 75 12
Articulating the differences between the
performance levels. 8 4.12 0 0 12 62 25
Discussions with other participants. 8 4.38 0 0 0 62 38
Providing additional details to the
performance level descriptors. 1 4 0 0 0 100 0
My expectations of students. 8 4.12 0 0 12 62 25 The difficulty of the test materials. 8 4.38 0 0 0 62 38 The student responses. 8 4.62 0 0 0 38 62 My experience in the field. 8 4.38 0 0 0 62 38 Discussions with other participants. 8 4.12 0 0 0 88 12 Cut scores of other participants. 8 4.12 0 0 0 88 12 Impact data. 8 3.62 0 12 12 75 0
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Would like guidelines of LOA to be present in future.
Efficiently done; leadership/guidance great help and support
Jake was great to work with!
Need more clarity on level of assistance- Scribe/- more info provide on how each teacher did
it.
I believe that we needed more information regarding the level of assistance. Some teachers
were not clear. Training for this may need to be more specific.
Appendix I—Evaluation Results 46 2010 Maine PAAP Standard Setting Report
APPENDIX J—POLICY ADJUSTMENTS
Appendix J—Policy Adjustments 1 2010 Maine PAAP Standard Setting Report
Achievement level Average
cut Standard
error Raw
Min
score
Max Percent of students
PD 61.1 NA 62 69 3.4 Proficient 35 NA 35 61 51.7 Partially Proficient 21.5 NA 22 34 36.9
PD 61.1 NA 62 69 14.1 Proficient 39 NA 39 61 41.5 Partially Proficient 21.5 NA 22 38 33.2
PD 61.1 0.4 62 69 20.5 Proficient 41.2 0.3 42 61 42.3 Partially Proficient 21.5 0 22 41 28.2
PD 66.9 0.6 67 69 33.2 Proficient 51.7 0.3 52 66 31 Partially Proficient 26.8 0.2 27 51 19.3
PD 84.8 1.2 85 99 10.1 Proficient 55.9 1.5 56 84 52 Partially Proficient 24.4 0.5 25 55 24.7
PD 91.3 0.9 92 99 19.8 Proficient 55.9 NA 56 91 53.1 Partially Proficient 24.9 0.6 25 55 18.8
PD 121.25 NA 122 129 9 Proficient 81.55 NA 82 121 35.4 Partially Proficient 32.45 NA 33 81 38.4 SBP NA NA 0 32 17.1
MDOE Policy:
Students in grades 2 through 5 have the opportunity to access tasks in levels of complexity 1–4.
Adjusted cuts were made based on the grade 4 cuts as stated above. However, when looking at the adjusted
cuts for grades 2 and 3, it was felt that the adjusted cuts at Proficient were too high.
MDOE reviewed 4–6 student samples around the Partially Proficient /Proficient cut points along with
participant rater sheets and determined that cut scores between Partially Proficient and Proficient for students
in grade 2 should be minimally lowered. The same process occurred for students in grade 3 keeping in mind
that grade 3 should be slightly higher than grade 2 but still minimally lower than grade 4. These new cuts are
reflected in the chart titled “Mathematics Policy Results.”
Writing grade 7 cut scores at Proficient was minimally lowered following the same procedures as
outlined above for mathematics.
Grade
Mathematics: Policy Results
2
SBP NA NA 0 21 8.0
3
SBP NA NA 0 21 11.2
4
SBP NA NA 0 21 9
5
SBP NA NA 0 26 16.6
6
SBP NA NA 0 24 13.2
7
SBP NA NA 0 24 8.2
HS
Appendix J—Policy Adjustments 3 2010 Maine PAAP Standard Setting Report
Achievement level Average
cut Standard
error Ra
Min w score
Max Percent of students
PD 22.0 NA 22 23 19.9 Proficient 14.4 0.1 15 21 28.6 Partially Proficient 9.6 0.1 10 14 22.4
PD 32.0 NA 32 33 15.0 Proficient 23.0 NA 23 31 31.6 Partially Proficient 11.6 0.1 12 22 39.8
PD 40.9 0.7 41 43 8.2 Proficient 23.8 0.6 24 40 33.6 Partially Proficient 12.1 0.4 13 23 41.8 SBP NA NA 0 12 16.4
Grade
Writing: Policy Results
4
SBP NA NA 0 9 29.0
7
SBP NA NA 0 11 13.6
11
Appendix J—Policy Adjustments 4 2010 Maine PAAP Standard Setting Report
Pe
rce
nt
of
stu
den
ts in c
ate
go
ry
0
20
4
0
60
8
0
10
0
2 3 4 5 6 7 HS Proficient with Distinction
Proficient
Partially Proficient Substantially Below Proficient
Mathematics: Policy Results
Appendix J—Policy Adjustments 5 2010 Maine PAAP Standard Setting Report
Pe
rce
nt
of
stu
den
ts in c
ate
go
ry
0
20
4
0
60
8
0
10
0
Proficient with Distinction Proficient Partially Proficient Substantially Below Proficient
4 7 11
Writing: Policy Results
Appendix J—Policy Adjustments 6 2010 Maine PAAP Standard Setting Report