11 2010 Winter Education Conference Consequential Validity Using Item- and Standard-Level Residuals to Inform Instruction.

11

2010Winter Education Conference

Consequential Validity

Using Item- and Standard-Level Residuals to Inform Instruction

2

Consequential Validity

• Districts must adopt instructional materials that are aligned to the Alaska Grade Level Expectations:– Districts have autonomy to select instructional

material.– What can a district do to evaluate the

effectiveness of instructional materials with the SBA?

3

Analysis

This presentation was designed to illustrate how the use of residual analysis could help districts and schools answer questions about the effectiveness of instruction. Commonly, the percent of students in each achievement level and the mean and standard deviation of scale scores are the only information available to describe student performance.

4

Topics

• What is a Residual?• How Residuals Control for Achievement

Levels• Interpreting Residual Plots• Item-Level Residual Plots• Standard-Level Residual Plots• Comparing Cohort Standard Residuals• Text Book Residual Plots

5

Introduction

• The description of the results of a test for a district or school, or gender, ethnic, or native language group can go far beyond reporting the mean and the standard deviation for the group.

• Looking at subgroups of items or students can often reveal interesting patterns that have important curricular implications.

• This type of analysis is called residual analysis.• This demonstration looks at the residuals for

selected districts and common textbook for two grades and is only a small sample of the types of questions that can be investigated.

6

What is a Residual?

• A residual is the difference between what you observe and what you expect.

• For each person/item interaction you can observe a 0 (wrong) or a 1 (correct). (Only multiple-choice items are used in this presentation.)

• Each student’s achievement is based on the total number of correct answers on the test.

• Each item’s difficulty is based on the total number of correct responses statewide.

7

What is a Residual?

• The student’s achievement level and the item’s difficulty combine to produce the expected score for that student/item interaction.

• The expected score will be between 0 and 1.

• This is easily interpreted as the percentage of times that the student would answer questions of this difficulty correctly.

8

What is a Residual?• If the student and the item are at the same location

on the continuum (construct), the student will have a 50% chance of answering the item correctly.

• If the student and the item are at the same location, the residual will be +0.5, if the answer is correct, or -0.5, if the answer is incorrect.

• This will produce a residual matrix for every student/item interaction (There are about one-half million per test/grade combination.).

• For Grade 7 Mathematics this would be a 56 column (items) by about 9500 row (students) matrix.

9

What is a Residual?

• The beauty of this matrix is that it allows you to sum any meaningful collection of rows or columns.

• For instance, you could combine all of the students into their respective districts. This would give you district-level results.

• Or you could combine all of the items that constitute the Estimation and Computation standard in Grade 7 Mathematics. This would give you a standard-level result.

10

How Residuals Control for Achievement Levels

• Because each residual is computed using the student’s achievement level, the resulting residuals are free from the distribution of student achievement.

• This means that comparisons across combinations of rows, like classrooms, buildings, districts, or demographic factors such as gender, ethnicity, or native language will be independent of the achievement level for that group.

• This property allows educators to use the residual matrix to answer important questions about the effectiveness of the curriculum at the standard level.

11

How Residuals Control for Achievement Levels

Person District School ESL i1 i2 i3 i4

P1 1 1 N +.3 +.4 +.5 -.4

P2 1 2 Y +.1 +.2 +.3 +.4

P3 1 2 N +.5 -.4 +.7 -.2

P4 2 1 N +.2 -.7 +.4 -.5

P5 2 1 N +.4 +.5 -.4 +.7

P6 2 2 Y -.4 -.3 +.8 -.1

This is an example of a small residual matrix.

12

Interpreting Residual Plots• The expected value for a residual or sum of

residuals is zero.• Positive values mean the student(s) are doing

better than expected.• Negative values mean that the student(s) are

not doing as well as expected.• The units of the scale are the proportion of

correct responses. An average residual +0.05 means that 5 percent more students answered correctly than expected. A -0.10 means that 10 percent fewer students answered correctly than expected.

13

Selection of Aggregates

Eight medium to large districts were chosen for analysis. They are used simply to illustrate the methodology.

Three common text books used in Grade 6 mathematics instruction were chosen for analysis. Again, they are used simply to illustrate the methodology. Not all districts use the textbooks in the same manner. Therefore, the results may not generalize to a specific district.

14

Item-Level Residual PlotsItem by Multiple Districts Plot

15

Practice Test Item Similar to Item 46

16

Item-Level Residual PlotsItem by Single District Plot

17

Item-Level Residual PlotsItem by Single District Plot

18

Item-Level Residual PlotsItem by Single District Plot

19

Item-Level Residual PlotsItem with Standards by Single District Plot

e = estimation & computation, f = functions & relationships, g = geometry, m = measurement, n = numeration, s = statistics & probability

20

Standard-Level Residual PlotsStandard by District Plot – Grade 7

21

Standard-Level Residual PlotsStandard by District Plot – Grade 6

Standard-Level Residual PlotsComparison of Grade 6 and Grade 7

22

23

Standard-Level Residual PlotsText by Standard Plot

24

What Next?

• What are some questions districts might ask after looking at residual plots?

• What are some district actions or next steps?

• What cautions need to be applied to the interpretation of the results?

25

Where can a district get more?