Setting Performance Standards Grades 5-7 NJ ASK NJDOE Riverside Publishing May 17, 2006.

Setting Performance Standards

Grades 5-7 NJ ASK

NJDOE

Riverside Publishing

May 17, 2006

2

What do we mean by “Standard Setting”?• Content standards: specify the content that

should be learned by students – given implementation of the state standards and an opportunity to learn.

• Performance standards: specify the precise level of performance required on an assessment to achieve some category or outcome. How much must students know and be able to do?

3

Proficiency Levels on NJ ASK

AdvancedProficient

PartiallyProficient Proficient

ProficientCut Score

AdvancedCut Score

100 - 199200 - 249

250 - 300

4

NJ ASK Grades 5-7 Standard Setting

Phase 1 20 panelists on the Math Grades 5-7 & LAL

Grade 7 panels 19 panelists on the LAL Grades 5-6 panels Used a Modified Angoff procedure to

calculate cut scores in a 2-day workshop Phase 2

Representatives from Phase 1 and state and local policy makers reviewed results

Examined consistency across grades and subjects

5

The Angoff Procedure• Research-based procedure used since

the early 1970s• The most commonly used standard

setting method — used in many other state testing programs as well as on certification tests

• Has undergone many modifications over the years and is often referred to as the Modified Angoff or Extended Angoff procedure

6

Angoff Procedure (con’t.)• Original Angoff procedure asked

panelists to examine each item on a test and determine whether a student who was just barely Proficient would answer it correctly

• A common modification is to ask “If we had 100 barely Proficient students in the room, how many of them would answer this item correctly?”

7

The Angoff Task

Review each item For MC items, determine how many out

of 100 just barely Proficient and just barely Advanced Proficient students should answer each item correctly

For OE items, determine the average score a just barely student at each performance level should receive

8

Validity Considerations The process was well-planned – see

standard setting proposal. All policy, operational and technical

decisions were reasonable and well-documented.

The judges were satisfied with the process and its results – see the evaluation results.

9

The Final Decision on Cut Scores – Reminder

Legitimate authority should have all the information they want or need to make an informed decision

At a minimum it should include: Overall results – Phase 1 and Phase 2 Variance in judgments and in measurement Impact on subpopulations in the state Impact on accountability Documentation of the process (with as

much detail as is necessary)

10

Ratings Cut scores are determined over three

rounds of ratings Ratings were made independently After each set of ratings, panelists

saw their cut scores and the average cut scores

There was a discussion between each round

11

Calculating the Cut Score Consider the MC ratings as probabilities We sum the probabilities to obtain a cut score for

the MC items and add the average scores for the OE items to get the total cut score

Item RatingPossible1 .85 12 .55 13 .40 14 1.5 3

Total= .85 + .55 + .40 + 1.5 = 3.3 points out of a possible 6 points

12

“Just Barely” Performance The idea of just barely performance

is key to the Angoff procedure We spent time talking about what it

means to be just barely Proficient and Advanced Proficient

Our discussions were driven by the NJ ASK LAL and Math Grades 5-7 Performance Level Descriptors (PLDs)

13

Phase 2 Meeting Policymakers and stakeholders gathered

to review the results of Phase 1 in light of NJ ASK program goals.

Reviewed results from Phase 1 along with standard deviation of judgments and impact data.

Compared results of Grades 3, 4 & 8 LAL and Math with Grades 5-7 LAL and Math and discussed the need for consistency across grades and content areas.

14

Phase 2 Results: Final Cut Score Recommendations Committee produced cut score recommendations for

NJDOE and the State Board:

ProficientAdvanced Proficient Proficient

Advanced Proficient

Grade 5 16 30 18 30Grade 6 20 32 17 31Grade 7 21 34 13 26

LAL Math

15

Evidence the Results Were Technically Sound

Survey Results (see handout) Standard deviation of judgments in each round:

ProficientAdvanced Proficient Proficient

Advanced Proficient

Round1 2.6 2.1 3.2 3.12 2.2 2.4 2.6 2.53 1.6 1.7 0.4 0.4

1 2.5 3.3 2.0 2.42 2.9 4.4 1.6 0.23 0.6 1.4 0.7 0.9

1 2.2 2.0 2.6 2.62 3.5 3.4 2.0 2.33 2.4 2.2 0.8 1.1

SD of Judgments

Grade 5

Grade 6

Grade 7

LAL Math

SD of Judgments

16

Distribution of Students by SubgroupLAL Grade 5

0%

20%

40%

60%

80%

100%

AP

P

PP

AP 8.1 6.7 9.6 11.3 1.8 2.0 19.0 10.0 1.0 0.6 2.0

P 70.7 70.3 71.3 77.4 59.6 62.4 73.0 76.5 50.2 42.0 60.2

PP 21.2 23.0 19.1 11.3 38.6 35.6 8.0 13.5 48.8 57.4 37.8

Total Male Female White Black H A Gen ED SP ED LEP EC Dis

17

Distribution of Students by SubgroupLAL Grade 6

0%

20%

40%

60%

80%

100%

AP

P

PP

AP 7.4 5.8 9.1 10.7 1.0 1.7 15.0 8.9 0.9 1.0 1.3

P 56.5 55.1 58.2 66.2 41.2 42.1 62.4 63.0 32.1 22.2 40.9

PP 36.1 39.1 32.7 23.1 57.8 56.2 22.6 28.1 67.0 76.8 57.8

Total Male Female White Black H A Gen ED SP ED LEP EC Dis

18

Distribution of Students by SubgroupLAL Grade 7

0%

20%

40%

60%

80%

100%

AP

P

PP

AP 5.7 3.9 7.5 8.1 1.5 1.2 12.2 6.9 0.6 0.5 1.2

P 56.8 54.3 59.5 67.3 39.9 41.8 69.2 63.9 31.0 18.8 38.4

PP 37.5 41.8 33.0 24.6 58.6 57.0 18.6 29.2 68.4 80.7 60.4

Total Male Female White Black H A Gen ED SP ED LEP EC Dis

19

Distribution of Students by SubgroupMath Grade 5

0%

20%

40%

60%

80%

100%

AP

P

PP

AP 22.6 24.0 21.2 29.4 7.7 10.2 48.9 26.7 7.0 7.6 9.5

P 56.0 53.5 58.7 59.1 51.6 54.8 45.1 59.2 44.4 40.9 51.4

PP 21.4 22.5 20.1 11.5 40.7 35.0 6.0 14.1 48.6 51.5 39.1

Total Male Female White Black H A Gen ED SP ED LEP EC Dis

20

Distribution of Students by SubgroupMath Grade 6

0%

20%

40%

60%

80%

100%

AP

P

PP

AP 13.4 14.4 12.6 18.1 3.3 4.3 32.2 16.1 2.5 3.0 3.8

P 53.0 51.6 54.6 62.5 36.6 41.5 54.4 59.0 28.7 25.5 38.0

PP 33.6 34.0 32.8 19.4 60.1 54.2 13.4 24.9 68.8 71.5 58.2

Total Male Female White Black H A Gen ED SP ED LEP EC Dis

21

Distribution of Students by SubgroupMath Grade 7

0%

20%

40%

60%

80%

100%

AP

P

PP

AP 11.8 13.5 10.2 16.7 2.2 2.7 30.6 14.4 1.4 2.1 2.5

P 47.6 46.5 48.9 56.9 31.1 37.2 52.2 53.7 22.0 24.7 32.5

PP 40.6 40.0 40.9 26.4 66.7 60.1 17.2 31.9 76.6 73.2 65.0

Total Male Female White Black H A Gen ED SP ED LEP EC Dis

22

LALPhase 2 Results

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Per

cen

t in

Lev

el

AP

P

PP

AP 4.4 4.4 8.1 7.4 5.7 8.0

P 78.8 77.2 70.7 56.5 56.8 64.3

PP 16.7 18.4 21.2 36.1 37.5 27.7

Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

Final Recommended CutscoresLAL Grades 5-7 (with LAL Grades 3,4 &8)

23

MathPhase 2 Results

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Per

cen

t in

Lev

el

AP

P

PP

AP 28.8 31.7 22.6 13.4 11.8 18.7

P 53.7 48.5 56.0 53.0 47.6 43.7

PP 17.5 19.8 21.4 33.6 40.6 37.6

Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

Final Recommended CutscoresMath Grades 5-7 (with Math Grades 3,4 &8)

24

Key Statistics from Surveys