English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate.

English Language Arts Performance Assessment:

Its Fairness and Predictive Validity

Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang

UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing (CRESST)

American Educational Research Association52.038–Applying Research-Based Performance Assessment

Models in Routine Practice in a Large Urban School District: The Pleasure-Pain PrincipleSan Diego, CAApril 15, 2004

Purpose

This study is to examine the English language arts performance assessment in the following two aspects:

1. Relative fairness

2. Predictive validity

Research Questions

1. Is the performance assessment a fair test?

2. Is the performance assessment predictive of the California High School Exit Exam (CAHSEE)?

Data Description

• 50 schools

• 5,427 students

• 9th graders in Spring 2001 and 10th graders in Spring 2002

School Description

Table 4. Means and Standard Deviations of School Level Variables (N=50)

Variables Mean Std. Deviation Average Class Size 27.19 1.63 Percent of Students in Lunch Program 0.56 0.21 School API Rank in 2001 2.88 2.09 School Enrollment Size (in 1,000s) 3.20 0.88

Student Description

• 21% of the students were English proficient;• 81% of the students were Hispanic;• 26% of the students were immigrant;• 75% of the students spoke Spanish at home;• 75% of the students were in Title 1;• 73% of the students were in lunch program;• 3% of the students were gifted; and• 1% of the students were in special education

program.

Hierarchical Linear Model(school level)

School level variables:

• Average class size• Percent of students in lunch program• School enrollment (in 1,000s)• API rank in 2001

Hierarchical Linear Model(student level)

Student level variables:

• Gender• Ethnicity• English language proficiency• Home language• Immigrant status• SES (lunch program and Title 1)• Special education program• Gifted status

Results on Fairness(Variance Partition)

Table 6Variance Component Results For SAT9 Reading, SAT9 Math and PA Scores

SAT9 Reading

SAT9 Math PA

SAT9 Reading

SAT9 Math PA

Level 1 - StudentProportion of variance attributable to students 93.4% 94.4% 92.3%

% variance reduced due to student variables 18.0% 12.5% 6.1%

Level 2 - SchoolProportion of variance attributable to schools 6.6% 5.6% 7.7%

% variance reduced due to school variables 65.3% 47.3% 9.8%

With PredictorsWithout Predictors

Results on Fairness(summary 1)

SAT9 Reading SAT 9 Math Performance Assessment

School-level VariablesSchool averageAverage classroom size% of students in lunch program

School API rank in 2001 Positive PositiveSchool enrollment size

Student-level Variables

Female Negative Negative PositiveEthnicity-Black Negative NegativeEthnicity-Hispanic NegativeEthnicity-Asian NegativeEthnicity-Other English language learners Negative Negative NegativeRe-designated Fluent E.P. PositiveHome Language - Spanish NegativeHome Language - Other

Immigrant NegativeFree/Reduced Fee Lunch NegativeTitle1 Negative NegativeSpecial Education Negative Negative NegativeGifted Positive Positive Positive

Results on Fairness(summary 2)

School level:• None of the 4 variables were significant

Student level:• Ethnicity and SES were not significant• Positive female effect• Positive gifted and former ELL effects• Negative ELL, Spanish (HL), immigrant,

and special education effects

Result on Fairness(school level)

Coefficients Effect Size

SAT9

Reading SAT 9 Math PA

SAT9 Reading

SAT 9 Math PA

School-level Variables

School average 22.38 36.58 1.80

(5.39) (5.33) (0.53)

Average classroom size 0.32 0.15 0.00 0.03 0.01 0.00 (0.19) (0.18) (0.02) % of students in lunch program 4.31 4.39 0.31 0.38 0.36 0.43 (2.47) (2.94) (0.21) School API rank in 2001 0.60 * 0.52 * 0.01 0.05 0.04 0.02 (0.18) (0.21) (0.02) School enrollment size 0.19 -0.03 0.01 0.02 0.00 0.01 (0.40) (0.45) (0.04)

Result on Fairness(student level 1)

Coefficients Effect Size

SAT9

Reading SAT 9 Math PA

SAT9 Reading

SAT 9 Math PA

Student-level Variables Female -0.80 * -2.10 * 0.10 * -0.07 -0.17 0.13 (0.26) (0.31) (0.02)

English language learners (ELL) -7.19 * -4.03 * -0.19 * -0.63 -0.33 -0.25 (0.72) (0.69) (0.04) Former ELL 1.25 0.90 0.10 * 0.11 0.07 0.14 (0.65) (0.70) (0.04) Home Language - Spanish -1.24 1.69 -0.12 * -0.11 0.14 -0.17 (0.85) (0.89) (0.05) Home Language - Other -0.48 2.91 0.02 -0.04 0.24 0.02 (1.77) (1.53) (0.07) Immigrant -0.11 0.15 -0.04 * -0.01 0.01 -0.05 (0.27) (0.41) (0.02) Special Education -10.04 * -8.73 * -0.42 * -0.88 -0.72 -0.58 (1.13) (1.72) (0.10) Gifted 11.67 * 14.84 * 0.40 * 1.02 1.22 0.54 (1.62) (1.72) (0.06)

Results on Fairness(Student level 2)

Coefficients Effect Size

SAT9

Reading SAT 9 Math PA

SAT9 Reading

SAT 9 Math PA

Student-level Variables Ethnicity-Black -5.32 * -5.57 * -0.06 -0.47 -0.46 -0.08 (1.09) (1.04) (0.06) Ethnicity-Hispanic -1.81 -3.35 * 0.07 -0.16 -0.28 0.09 (0.99) (1.40) (0.06) Ethnicity-Asian -3.82 * 3.16 -0.05 -0.33 0.26 -0.07 (1.22) (1.90) (0.09) Ethnicity-Other -1.65 0.33 0.06 -0.14 0.03 0.08 (1.31) (1.90) (0.07) Free/Reduced Fee Lunch -1.29 * 0.41 -0.04 -0.11 0.03 -0.06 (0.39) (0.45) (0.02) Title1 -1.87 * -1.83 * -0.05 -0.16 -0.15 -0.07 (0.69) (0.77) (0.03)

Results on Predictive Validity(cross-tabulation)

Figure 1. Percent of Grade 10 Students Passing CAHSEE (ELA) As Predicted

By Their Grade 9 Performance Assignment Scores (N = 5,427)

37.5%

56.4%

75.3%

92.2%

0%

25%

50%

75%

100%

Not proficient Partially proficient Proficient Advanced

2001 Performance Assignment Scores

Pe

rce

nt o

f Stu

de

nts

Pa

ssin

g th

e

20

02

CA

HS

EE

Results on Predictive Validity(school level)

Table 8

HLM Results on Performance Assessment's Predictive Validity

Coefficient Std. Error Log Odds

School-level Variables

Intercept -5.664 * 1.169 0.00

Average Class Size -0.009 0.040 0.99

Percent of Students in Lunch Program 0.103 0.389 1.11

School API Rank in 2001 0.023 0.037 1.02

School Enrollment Size 0.215 * 0.081 1.24

Results on Predictive Validity(student level)

Coefficient Std. Error Log Odds

PA 2001 0.385 * 0.042 1.47

SAT9 Reading 2001 0.123 * 0.006 1.13

SAT9 Math 2001 0.021 * 0.004 1.02

GPA 2001 0.365 * 0.067 1.44

Female 0.382 * 0.080 1.46

Ethnicity-Black -0.745 * 0.304 0.47

Ethnicity-Hispanic -0.394 0.313 0.67

Ethnicity-Asian -0.002 0.209 1.00

Ethnicity-Other -0.117 0.255 0.89

English Language Learners -0.397 * 0.121 0.67

Re-designated Fluent E.P. 0.369 * 0.108 1.45

Home Language - Spanish -0.188 0.162 0.83

Home Language - Other -0.278 0.296 0.76

Immigrant Status -0.118 0.076 0.89

Free/Reduced Fee Lunch -0.174 0.091 0.84

Title1 -0.129 0.120 0.88

Special Education -1.493 * 0.381 0.22

Gifted 0.152 0.193 1.16

Results on Predictive Validity(reading)

Figure 2:Probability of Passing CAHSEE by SAT9 Reading Scores

0%

20%

40%

60%

80%

100%

0 20 40 60 80 100

SAT9 Reading Scores

Pro

bab

ility

of

Pas

sin

g C

AH

SE

E

Not Proficient

Partially Proficient

Proficient

Advanced

Note: The calculation w as done assuming the student to be a non-immigrant male White English-proficient student w ho spoke English at home, paid for lunch at school, w as not classif ied as Title 1, gifted, and special education, and scored at the mean level in GPA and SAT9 mathematics test w hile enrolled in a school w ith mean school characteristics.

Results on Predictive Validity(mathematics)

Figure 3:Probability of Passing CAHSEE by SAT9 Mathematics Scores

0%

20%

40%

60%

80%

100%

0 20 40 60 80 100

SAT9 Mathematics Scores

Pro

bab

ilit

y o

f P

assin

g C

AH

SE

E

Not Proficient

Partially Proficient

Proficient

Advanced

Note: The calculation was done assuming the student to be a non-immigrant male White English-proficient student who spoke English at home, paid for lunch at school, was not classified as Title 1, gifted, and special education, and scored at the mean level in GPA and SAT9 reading test while enrolled in a school with mean school characteristics.

Conclusions

1. The performance assessment a fair test.

2. The performance assessment is predictive of students’ California High School Exit Exam results.